Is it possible to "grow out of" a Kirby CMS?

Thanks for the reply.

Kirby provides a database class in its Toolkit so even the folks that build Kirby understand that a DB is best in some cases. In almost all of those cases - where a database is best - the client will need to manage the content in the database. I would prefer if the client could use Kirby’s panel as opposed to jumping to some other interface. As far as the clients knows, it’s just plain odd.

Straight up… I need 1000 pages under 1 page. Is this an issue or not? Yeah… it’s gonna be on a SSD drive. No. the client will never organize things into any years (come on - what do they know or care about such things?). I don’t necessarily want to use a db, but I’m just trying to do what’s best for my clients. Every site grows over time.

Well, the Panel obviously only works with files, not with a database. The only way you could maybe achieve this, is with a plugin that connects to the database and then create custom Panel views where the user can edit the database data.

Is 1000 that many? We run page with 500 pages where there is lot of relation between pages (looking for multiple related pages so kirby has to go trough lot of tags stuff like that). It was always very fast. I the only problem is usualy the search - where db can do lot of work for you and does things that would be impossible just in php.

What i am trying to say - are you sure that its the filesystem that makes your site slow? People underestimate the filesystem but they are damn fast and DBs are actualy working in very similar ways. Don’t forget once files are used they end up in memory - you dont read it from disk all the time. Its same like with databases. Also how big is the real content (the text files) - 50MB? Thats nothing in memory.

Most of the time websites are slow because of bad frontend code. Maybe you have too little ram on your server (or you have overcrowded shared server). Or you have little bug in your template that is making things slow.

I would try to be really sure about the cause before switching to db. You might end up with site as slow as before.

If its live public site, you might aswell post url here. People might have ideas about the frontend code :slight_smile:

Thanks Krisa (and Sonja) for your time.

I really don’t want to use a db if Kirby can handle upwards of 1,000 “pages” without performance issues. I have personally never run any tests on massive amounts of pages. Answers in the forum here don’t seem definite regarding large amounts of pages and performance. I suppose that’s because it all depends, but I’m not creating anything crazy here. Consider 500 product pages, each with 5 product shots. I need to search the products. I need to categorize the products (note: 1 product may live in multiple categories). Pretty common stuff.

Based on your comments I’m optimistic that using Kirby is still a good option.

1 Like

I’d do a test, create 1000 pages as subpages of a single folder programmatically, including images, and see how it performs, both on the frontend with searching and filtering, as well as handling that many subpages in the Panel. With a fast server and SSD, up to 1000 pages should in fact be doable from what I’ve read here in the forum. But if it’s about product pages, I wonder if it wouldn’t make more sense to categorize them into subfolders first instead of putting them all into a single folder (yeah, even if they have multiple categories, there are usually ways to sort them into buckets in a way that makes it easier for the client to handle these products as well).

I think it will be just fine.

Like i said the problem really is the search. People are used to very good search engines and it can be quite advanced work thing to achieve that. Kirby search is “dumb”, it works for basic things but you will definitely need things like ranking and stemming and if you are not english - you will need it even more.

The way i solved it might sound counter intuitive but i am using both db and filesystem. I am basically indexing all the pages specifically for the search and use db just for the search. This way the files are always source of truth and db is just replication (besides i index only stuff that needs to be searched). I tried to go the only db route but ive found it to be worse solution. I didn’t seem to get better performance out of it and i lost the power of the filebased cms (easy to debug and backup etc.).

Yes you are duplicating content but redundancy is good. Few megs of hard drive space is cheap.

Its also easy to achieve - you either index on panel hook when somebody edits the text file or just do cron that will update the db from time to time (i do this, because the data don’t change that much).

It has huge upside if database fails which they tend to do from time to time - you are fine, your site is going you just don’t have search.

This design worked so well i am actualy considering streamline it (do plugin for it) next time i will be doing site like that.

if the created posts contain a date (which i presume is true) you could create an automated archive function which creates the archive folders and moved content from anything but current year into its proper place.
this could also be done with a hook and dir::move.

in the frontend you just have to filter by template or iterate all archives and current.

1 Like

i do not understand why the default search is “dump”? it even has scoring based on matches.

about creating a plugin… unless yours will be free the algolia plugin by kirbys @lukasbestle seems very promising and does excactly what you just describe, right?

Thanks Bnomei.

Having pages that are organized by date moved automatically to an archive, based on year for example, seems like a practical solution to keep things tidy and preforming well. I could see this method used quite often. Hmmmm.

With files moving I can imagine how that could break some queries if you’re not careful, but I guess that why you mention filtering by template as that’s pretty solid. Interesting.

When moving pages to an archive, make sure that old URLs are rerouted after moving, otherwise you end up with lots of invalid URLs.

Not sure if this helps with feeling confident in kirby’s performance but it it’s worth a look.

1 Like

Well if you try Kirbys search is just very basic. The mainstream databases have years of time put into making search good. When you start to get to territory of “scott white mountain bike” - when you are searching in titles, tags, anotations and fulltext together - you can’t setup ranking in good way. It also has problems if you are using accented characters and all kinds of other details.

Algolia plugin is exactly that, infact algolia is most probably going to give much better results than your database is ever going to - you would have to use something like elasticsearch or solr to get to that (and spend time setting it up properly)…

BUT - try to bill your client 50usd / month (600 a year) just for search when his last site had perfectly fine fulltext search done using mysql. It might be worth it to ecommerce (better results = more revenue) but on regular sites i guess not.

BTW i am not complaining, kirbys search is enough for lot of things. I don’t think there is any reason why it should be better - search is always going to be separate dedicated thing.

2 Likes

Can you elaborate on this a bit?

flat hirachy
panel subviews slow down if folder contains too many subfolders. for 1000 pages better create 20 x 50 than put 1000 in one folder. the panel subviews parse each contained folder – the more the slower. see more here.

site()->index()
this function gets all pages you have in content. this is not fast if you have a lot. usually you do not need all but a filtered set of them. if you have lots (like 500 or more) pages it might be faster to iterate of the childs yourself and use filterBy() etc. pseudo example…

// get all 20k exposes nested in 10k projects (root: project):
$allProjectMaybeSlow = site()->index()->filterBy('template', 'expose');
$allProjectsFaster = array(); // array
foreach(page('projects')->children()->filterBy('template', 'project')->children()->filterBy('template', 'expose') as $expose) {
   $allProjectsFaster[] = $expose;
}
$allProjectsFaster = new Pages($allProjectsFaster); // Pages Collection

cache
if your content is static you could remove the impact of site()->index() by turning on caching. or just cache the result of your filter-query.

root of all evil
like always measure first and optimise only if needed.

2 Likes

about Algolia. as far as i can see from their pricing table they offer a free community plan for 10k records and 100k operations. so its not 50 bucks or nothing.

This is only useful if your clients can live with the Algolia logo on their site, though. The pricing seems to be around 50 bucks for all these services, it seems, not only Algolia.

As regards performance, I created a folder with 1000 subpages yesterday, with three images in each folder. Filtering these pages on frontend and searching seems really fast, the Panel lags a bit when accessing the parent, but still fast enough, I think. Will test this a bit more with more images, more content and more HTML after my vacation.

2 Likes

I just wanted to add my 2 cents to this :slight_smile:

I run my personal blog on Kirby, but since I’ve gone Indieweb, I’ve imported all Twitter and Instagram posts I ever posted. Recently I started to use Foursquare/Swarm, so I’m adding about 6 posts a day. At the moment of writing I have 8884 posts, in a folder structure with one folder per year, with folders per day of the year, with folders for every post (YYYY/DDD/N, I’m rewriting URLs to get /YYYY/MM/DD/N/optional-slug). On top of that I have views for only blogposts, photo’s, checkins, and every category/hashtag is searchable.

I have to admit that I got rid of the panel, but that is because I don’t like that it leaves empty fields written out in my content files. (I have a lot of optional fields.) But I do think my site could run with the panel. It might need one panel.page.create hook, but that’s all.

I use Kirby’s standard folder structure, but index it with a database. The database is just an index, so I can get rid of it at any time I feel like and regenerate it. The main content is stored in the .txt-file and if you visit a post-permalink, I don’t access the database at all. The database is only used for the feeds and the category/tag-searches.

Because of all my optional fields, I have quite a lot of template-logic, so I cache the HTML-files of the ‘entry’ snippet. (one permalink version and one feed version) The rest of the page is still rendered without cache, it’s just the entry part (the white block in my current design), and it’s still fast enough to do the caching on the first load after update. I use this caching moment to update the database/index.

So I do the following on page-load:

  • if there is a .html version, and the f::modified() of the .txt version is older, and the snippet is also not modified
    • return the cached html (stop here)
  • if the is no .html version or the .txt was modified
    • update the index/db
  • generate a new .html version from the snippet and return that.

This way I can update via Micropub, via the Panel, via $page->update() or even via FTP, and the db will be updated on next visit of that page.

The only caveat is that the post will not show up in the stream if no-one has visited it yet, so I have added an indexer::add($page) to my Micropub endpoint, and would have to add one on the panel.page.create (and why not also update) hooks. When updating via FTP I have to make sure to visit the page.

All in all, now my site is fast, and still has the benefits of Kirby’s very readable file system.

Please shoot criticism my way now :wink:

3 Likes

Can you elaborate on what information is saved in your database? Just pointers to URLs?

It’s quite dirty at the moment, not a real example of how one should be handling this, but it’s an SQLite database (using Kirby’s database class) with two tables: page and category.

Page has id (in the format 2017/179/1, which is $page->id()) and a couple of fields I like to track: lang, visible, private, published(date), (post)type, deleted.
Category has id, page_id and ‘category’, which is just the string. (And that’s not how one should do it, I know :wink: )

Maybe I can work it out in a proper plugin, but this is the file I keep in my plugins folder at the moment: https://gist.github.com/sebsel/53660bc8c3f7f2f31368cc0007d9d71e

1 Like

you could add that you have created a plugin for using micropub with kirby.