Kirby3 Boost - makes loading a lot of pages in one request fast + unique ID

[bnomei]

i created an issue SQLite3 object has not been correctly initialised or is already closed · Issue #1 · bnomei/kirby3-sqlite-cachedriver · GitHub
lets continue discussing the sqlite driver issue there.

[JonasHolfeld]

Thanks for the fix!

One more question: does the Boost-plugin also speed up the Kirby query-language inside a blueprint? Like when used for my many-to-many plugin:

foreignkey:
        label: Project
        type: multiselect
        min: 1
        max: 1
        options: query
        query:
          fetch: site.find('projects').childrenAndDrafts
          text: "{{ page.title }}" #<-- will this be quicker with boost?
          value: "{{ page.boostid }}"

Will this speed up the fetching of the titles?

Thanks

[bnomei]

yes. once you have a page boosted accessing any content field will be faster since the initial load was made from cache.
in your example the title will be loaded the same time as the boostid. but that load will be from cache and thus faster than from disk.

it does not however reduce computational load like from using site->find or site->index. you would need to create a statically cached collection/sitemethod for these. static because it might be called by kirby more than once in a single request. i will publish a short example later.

[bnomei]

with boost its quiet easy to measure/see the performance gain. just toggle the debug mode. boost will always write to the cache but only load in non-debug mode.

[JonasHolfeld]

Thanks, thats very helpful. I will make a new release of my plugin using boost then soon.

[bnomei]

i explained a bit how to create static cached collections here:

Remember this - Caching in Kirby

How to make blueprints with query in multiselect option with fetch faster? The problem with fetch Lets imagine articles where we want to reference related articles by their autoid or boostid. site/blueprints/pages/article.yml # other fields ... category: label: Related articles type: multiselect options: query query: fetch: site.find('articles').childrenAndDrafts text: "{{ page.title }}" # boosted value: "{{ page.boostid }}" # boosted In your frontend you do something…

[bnomei]

New version 1.8.0 adds a site index crawler with lower memory footprint than core kirby. This makes it possible to run a callback on a huge amount of pages.

Using site()->index() in Kirby will load all Pages into memory at the same time. This plugin provides a way to iterate over the index with having only one page loaded at a time.

$boostedCount = 0;
$indexCount = \Bnomei\Bolt::index(function ($page) use (&$boostedCount) {
    // do something with that $page like...
    $boostedCount += $page->boost() ? 1 : 0;
});
// or just
$boostedCount = site()->boost();

Within some limitations you could use \Bnomei\Bolt::index($callback) for your own code as well like when counting or updating pages.

[bnomei]

The new version 1.9.0 add support for content caching of files and users. So if you have to read the content files of many of them then do consider using the Boost plugin to speed up the load time up to 3-4x times.

[bnomei]

I update my boost plugin demo server to most recent version of kirby and kql. also moved it to a “slow” 5 euro/month hetzner instance to better show how much the plugins speeds up content loading.

the main demo page shows a general read/write benchmark for caches but that needs to be taken with a grain of salt since when using them with boost there are lots of initial writes (using transactions to batch them) when indexing and later on there are just a massive amount of read calls. some caches like sqlite and redis really excel at read speeds.
https://kirby3-boost.bnomei.com/

the subdomains each show a cache driver like apcu, redis, sqlite, mysql and php. the null cache driver is kind of special and at the same time not… it simply is running no cache at all. use this to compare how much faster things get using boost and a cache driver.

read 5500 pages in one request => earthlings
apcu: 1300ms
sqlite: 1500ms
redis: 1700ms
php: 2500ms (that loads all 38k pages of the demo since its a mono cache file)
mysql: 3300ms
null: 18.000ms

kirby actually has to load more than 5500 with all parent pages in the tree.

read 20.000 pages in one request => karma of humans
apcu: 14.000ms
sqlite: 14.000ms
redis: 17.000ms
php: 22.000ms (not sure why thats slow actually, will check it out, should be around 14-17sec)
mysql: > 30.000ms (thats 20k or more queries, no optimization yet)
null: 22.000ms

with lots of pages the performed computations of the demo take up a major part of the response time. the null driver performs well enough here but thats partly due to boost resolving the page uuids to page objects and not the core logic (see toPagesBoosted).

summary: as long as the read speed of your cache is faster than reading from files you gain some performance boost.

[carstengrimm]

quick question:

lets say i have product pages which update their price every 30 minutes (e.g. page->price())

the cart itself collects the uuid and qty to calculate the cart array/session with the latest price

in the calculation if i were to use boost, it seems as though also the price is being cached. might that be the case? i thought boost is mostly speeding up the page lookup but not the contents?

because in my local demo i could see the prices do not add up and a certain price seems to be stuck/cached even after a price update.

within kirby’s methods for caching (tried apcu and sqlite) i make sure to output everything via post requests which should somewhat avoid caching - the same for disabling cache. if i were to remove boost from the calculation functions, it seems to calculate just fine again.

 // EXAMPLE
function cartTotal(){
    $total = 0;
    $cart = getCartSession();
    if(!is_array($cart)) return $total;
    foreach($cart as $item){
        if($page = page('page://'.$item['uuid'])){ // here using boost($item['uuid'])
            $total += $page->price()->value() * $item['qty']; 
        }
    }
    return ($total);
}

Edit:
Seems from the first post, it caches the content as well.
To explain further,

There’s a price countdown which updates the page()->price() every half an hour. The update itself obviously seems to work, but it doesn’t seem to flush boost cached / or update boost content of each certain page

adding page()->boost() after the actual price update, does not seem to make it work in the calculations…

[bnomei]

@carstengrimm

I thought boost is mostly speeding up the page lookup but not the contents?

boost is about content caching first and also resolving relations between pages. it kind of was a successor to autoid until the uuids became core.
see readme:

Kirby3 Boost
⏱️ up to 3x faster content loading
🎣 fastest page lookup and resolution of relations

what boost does is taking the modified timestamp of the page into consideration and maybe load the content from a cache.
it will write on any content change. so calling page->update() to save the new price should force boost to write a new cache for that models content as well.

bnomei/kirby3-boost/blob/76c51b5e2c82f0800a3b7efefa610ed25da27092/classes/ModelHasBoost.php#L131

it should not make any difference if you use the page or boost helper in your example on what content gets loaded. the boost helper just resolves the object in a different way. it does not alter how it gets loaded. the later is based on the model (extending the ModelHasBoost trait).

you can check if the ModelHasBoost is the issue by setting the config values to false. 'bnomei.boost.read' => false and 'bnomei.boost.write' => false

is the cartTotal function within a cached call - not related to boost?

i assume you are using k4? its a bit hard to debug this wihtout seeing the whole setup. you can invite me via github or PM me some more code? I will sign a NDA if you need one.

[carstengrimm]

• kirby 3, latest
• the cartTotal function is within the plugin index

Basically using fetch JavaScript API, i’m pulling /cart.json via a route, which outputs html (via snippet) which will be loaded from the fetch and added to the cart div with innerHTML…

the cart contents are output correctly, e.g. going though the array, correct qty, but the prices seem not always be uptodate with what’s saved in the page (if i check via panel)

The fetch request includes no-store
The route shouldn’t be caching as it’s including a session (kirby docs), i have also changed to get the cart.json via post

That’s whats saved in the product-page:

That’s what’s loaded via fetch>route>snippet which uses the cartTotal function

[carstengrimm]

@bnomei

'bnomei.boost.read' => false and 'bnomei.boost.write' => false

boost read/write “true” (default)

boost read/write “false” via config

[bnomei]

it the content file it says 59651,61 but with boost r/w to false it gets 59628,14?

here is a catch to consider. since boost is using the default kirby caching logic all keys are prefixed with domain. so if your frontend is www.domain.com and your api at api.domain.com you will get invalid results because the api is reading from its cache. and the panel behind www will read another one. and boost has no way of clearing both caches (for now).

[carstengrimm]

i am using regular $page only, no access to the api,

inside the content file:

----

Price: 59638.42

----

boost read/write to FALSE

if i turn it back to TRUE, the Sum will be

59,368.33 € (not correct)

(the prices obviously changes, still overall the calculation seems to be cached somewhere as it’s stuck to the previous screenshots example.

EDIT:

BTW. I addded the models ModelHasBoost

to the product page template, i guess that should be sufficient to get the correct price everywhere using a route?

[bnomei]

its really hard to debug this remotely.

i am confident that boost has no major issues here so it must be a semi broken caching state. like boost not setting the modified timestamp when the price changed for some reason.

also boost stores the modified dates of content files only on hooks (by default, can be changed in config)… so if you edit a file in the editor it wont know about that.

please try calling a page->update on the price and vardump the value of the price before and after that. also add vardumps to ModelHasBoost methods for cache read and write to see whats going on.

since you are using an api vardumping is not ideal. maybe you could log these to a file or use a tool like rayapp.

[bnomei]

I think i found a possible case where an invalid cache might be created.

:bug: possible invalid cache state

It happened if the modfied timestamp could not be retrieved. It kept the old cache. Now it will clear the old cached value instead.

I will create a release next week. but you could test it… its one line.

To clear the currently invalid cache for your page call ˋ$page->deleteContentCache();ˋ once

[bnomei]

That state where the modified timestamp could not be read happened to me in my tests as well thus the if clause but its hard to replicate. Multiple file access in rapid succession, locked content file from kirby (?), not flushed timestamp on harddisk level…
But the fix should at least avoid invalid states now. Thanks for pointing out that issue.

[carstengrimm]

my problem still occurs. i thought after clearing the cache the problem is resolved, but it didn’t.

[bnomei]

i am out of my wits. unless you send me a reproducable example i dont know how to help you here. you could also contact me via discord and we could arrange for a screenshare if you dont want to send me any code.