A digital library of 50,000 books

I’m about to develop a new project: a digital library of (more or less) 50,000 books. And I would like to develop it with KirbyCMS (of course) and Algolia.

I know that Kirby can have some performance issues where there are more than a few hundreds of subnodes under the same node. So I want to store the books sorting them by author, with the following structure:

  • books
    –A
    —A-AE
    —AF-AM
    —AN-AZ
    ----Asimov Isaac
    -----I, Robot
    -----Foundation
    -----Black Widowers

As the english alphabet consists in 26 letters, breaking the structure down of every letter in 3 (or maybe 4) subnodes (A-AE, AF-AM, AN-AZ) I will obtain 263 nodes. 50,000 / (263) = 641 average nodes, which could be accetable (still many indeed, but the logic scalable).

I know that with so many items, a part from the structure, I could run in a lot of issues with Kirby native queries. But here is where Algolia comes to help: I will create several indexes to display the most common views (“the latest 50 books added to the library”; “the most read books in the last 30 days”; …ecc), and of course I will use Algolia as a search engine.

Am I completely crazy or it sounds good?

1 Like

There is a topic on this question, couldn’t find it just now. However if you are willing you could try it out I guess.

https://getkirby.com/docs/cheatsheet/pages/create - using the create()-Function in a loop and create your 50.000 pages, then see how it works.

I have a couple of ideas but I need to better understand where bottle necks might happen.

  1. How you will be interacting with these pages?
  2. Is this library something you will feed into Kirby and forget, or will it be updated frequently on the Panel?
  3. You already mentioned searching with Algolia, which is great, how will you categorize the books on the front-end?
  1. How you will be interacting with these pages?

Four ways:

  • Users can view the single book card (classic kirby-powered query)
  • Users can open the flipbook (pdf viewer) and in certain cases they can also download it (same as above)
  • Users can search through the whole sites (using algolia)
  • Users can access to specific views as “the most downloaded books in the last 30 days”, or “all the books of sci-fi genre”, or “all the books of the same publishing house” and so on. For these kind of queries I don’t know what is better: kirby-powered queries, or algolia, or static files (updated every hour or so).
  1. Is this library something you will feed into Kirby and forget, or will it be updated frequently on the Panel?

The second one

  1. You already mentioned searching with Algolia, which is great, how will you categorize the books on the front-end?

By author, genre, publishing house, phisical library where the book is from, language, subject, … 10-15 attributes I guess.

After doing some testing for myself I think it’s important to get the pages structure right. Kirby is fast even with a couple hundred pages on the same directory.

The total number of pages on the site doesn’t affect Kirby for things like these. It yields pretty good results.

I totally agree on Algolia for searching. For the other sections Kirby cache can handle them well.[quote=“hvsrmusic, post:4, topic:6356”]
Is this library something you will feed into Kirby and forget, or will it be updated frequently on the Panel?

The second one
[/quote]

The fact that the panel had such a great performance with ~ 65K pages totally blew my mind. I didn’t expect it! You can try for yourself.

Algolia is the way to go for this, specially if users can combine multiple filters at the same time. If you can do that directly from user browser to Algolia’s server with Javascript that would be even better. I used realtime search with Javascript for a client and Algolia’s server responds in < 80 ms. That would increase considerably if I were to put my server in between.

Please share the results you got when you are done with this project!

Personally I would use the panel for custom pages and if the 50K books are generated somehow I would put that into a database.

https://getkirby.com/docs/cookbook/database
https://getkirby.com/docs/developer-guide/toolkit/database

1 Like