Panel - How to switch file system to a database

Kirby 2 Suggestions šŸ’”
1

Iā€™ve been having this idea for about a year. The idea to switch the Panel file storage system to a database.

Use 20.000+ pages in the Panel

In some cases when we have 20.000+ child pages with the same parent, with categories, tags, search etc. Itā€™s commonly asked if Kirby is still the way to go. Most of us say yes, Kirby has database support. Then how can we use the Panel for it? Short answer, we canā€™t. It would take too long to load, especially the page listings.

Using a database in the panel

Iā€™ve been thinking of the idea how Kirby could implement a database solution in the Panel. If Kirby is going to set up a databas structure, it would not be optimal for all sites, because every database structure is very different. Kirby could never get it right. One database structure does not fit all, if we want the real strength of a database, performance.

Solution

To not have Kirby set up a database structure, we need to figure out a way to be able to do it ourselves. I think being able to register three components could solve the problem.

PanelListing component

When using this component we could change the listing of the children of a page. Instead of list pages from the folder structure we could populate it from the database.

PanelPage component

This component could be used to present the data into the Panel page form. Send it as an array would probably be the best. Then the panel can just present the data in the form.

PanelSave component

On save if we have this component it could hijack the save function and let us to what we want, save data to the database in this case and trigger a success or an error.

There are probably some pitfalls that I did not think about, but so far I like the idea to be able to control the data in and out of the panel. These components could probably be used for other things as well.

2

I have few questions:

  1. What is the performance bottleneck when having 20 000 child pages? Is it really the filesystem?
    Lot of the pains i have seen people to have here are connected to panel. It is hard to work with 20 000 pages but it is as hard as working with 20 000 records in database. It doesnt change the situation. I believe lot of those problems could be solved using different page listing field in panel. In similar ways like kirby-sortable but it would have to be optimized for showing and filtering many pages. DOM is your enemy here. It is actually not php that canā€™t render html page with 20 000 records fast, it is your browser that canā€™t show 20 000 DOM nodes fast. Luckily this is a common problem and there are many JS solutions that would be easy to use (basicaly in dom you have only things that are in viewport or are about to enter).

  2. Instead of replacing the filesystem with database completely, wouldnā€™t some smart sync tactic be better solution?
    When you start to have everything in db, you introduce yourself to new set of problems that kirby is not made to handle. I know lot projects with bigger scale do exactly that. They might have wordpress as administration interface but rendering of client pages is handled by something else and their content from wordpress database is synced with something very fast like redis. I mean this is overkill but what i am trying to say - you can think of kirby core and panel as completely separate thing in same way as you could think about your db. Replication is good thing. You can

  3. If i understand correctly you want database as faster alternative to files. This is in fact already in kirby - its called cashing. If you believe that filesystem is the bottleneck - it would be much better idea to start with proper cashing using something like memcached or redis. You might solve your problems without much work.

From my personal experience - i donā€™t believe filesystem is that much slower than a database. The bottlenecks are in different places. We also help our clients with wordpress sites. WP uses mysql and the sites with same scale are much slower than kirby. Its because of amount of wp code and all kinds of trash there is.

Database will make kirby sites faster if you have big amount of pages and need lot of dynamic stuff (like search or pages put together from 20 other pages). But if you do page(ā€˜path/to/mypageā€™) or you query the same page the difference will be really minimal - no matter how many pages you have.

But maybe i am wrong, i just havenā€™t seen any good evidence yet. I am a bit worried that it is this myth ā€œdbs are fastā€ retold again and again.

1 Like
3

.1. What is the performance bottleneck when having 20 000 child pages? Is it really the filesystem?

Yes.

As I wrote, in this example we want the pages in the same parent. For example we want to get all the pages that has the category ā€œCarsā€ or the tags ā€œBigā€ and ā€œRedā€, or a part of the title is ā€œVolvoā€. In this case it needs to read the files in order to get the information. In fact, it needs to read all of the 20 000 pages to figure it out.

In a database we can index colums and we can use IDs and build relationships between the tables. In short it means it does not need to read all the data to find out which pages to list. It will only need to read the data that is needed for the task.

DOM is your enemy here. your browser that canā€™t show 20 000 DOM nodes fast

Without a pagination it would be a major impact on browser performance, but with a pagination it would be fine.

.2. Instead of replacing the filesystem with database completely, wouldnā€™t some smart sync tactic be better solution?

I donā€™t really know what you mean in this case, or how this could be a solution. Maybe tell about more how you think it could work with the Kirby Panel?

.3. If i understand correctly you want database as faster alternative to files. This is in fact already in kirby - its called cashing.

Yes, speed is the reason. Caching is great in many cases but letā€™s say we want an advanced site search for our 20 000 pages with the possibility to filter by categories and tags as well. Then the cache would in many cases not work, because every new search would be a new request for the cache to handle.

i donā€™t believe filesystem is that much slower than a database.

It depends on your data. For small sites Kirby is very fast, probably faster than it would be with a database.

But maybe i am wrong, i just havenā€™t seen any good evidence yet. I am a bit worried that it is this myth ā€œdbs are fastā€ retold again and again.

I donā€™t have any proof, but that databases are really good with big amount of well structured data itā€™s not a myth.

To have a real example we can say that itā€™s an e-commerce site with many categories, tags, price groups, sizes etc. It contains an advanced search and 20 000+ or 100 000+ products.

I started a test with Kirby and a database (not in the Panel) and itā€™s this site: https://modehallen.se. It has only 200 products and no search but anyway, itā€™s a test so far and proves that Kirby works well with a database outside the Panel.

4

I dont think it has to read all the files from disc. The filesize will be what? 50 megs? It will read it from memory.

But we mean the same thing. I had to tackle similar thing. By sync i mean replicating data from your content to DB. It becomes kind of cashing layer above your site but allows the dynamics of database. You leave filesystem as source of truth and replicate it to your database. You donā€™t need to replicate everything, just the stuff that you need. In your templates you just read from db afterwards.

Its suprisingly easy really and i had lot of success with it. Only thing to take care of is how you replicate to your db. This can be done after panel updates or you might be smarter and make simple queue just write somewhere what should be updated and have some cron script to take care of it from time to time (it makes panel bit faster). I am actually just bruteforcing it and flushing/updating the data every morning (for my usecase it was good solution). You can also make widget button that does this, rebuilds everything if something is wrong.

For me it was better solution than go full DB for many reasons. Added bonus is that if you canā€™t connect to db you can display data from the old filesystem :)) (except search).

5

You are both right. The files are most likely cached in memory by the filesystem. But Kirby still needs to parse all files. The Kirby text files are indeed not meant to be searched through on very large sites.

However I agree with @krisa that replacing the file system with a database leads to all sorts of problems Kirby was intended to solve.
Search can be implemented with a service like Algolia or an index DB very well. There is no real need to replace the whole storage system just for search.

Please keep in mind that introducing such an API would also create a lot of edge cases weā€™d need to test. I donā€™t think this is worth it. If you really need a database for a site, use a database-driven CMS. :slight_smile:

1 Like
6

Fair enough. At least we now have had a discussion about it. :slight_smile: