Unfortunately, this query causes a delay in loading the panel of ~9s (to the document-fields endpoint) which the client has really begun complaining about significantly. This is occurring locally on an 8-core i9 MacBook Pro with 32gb RAM and a digital ocean droplet with significant resources (4cpu, 8gb RAM)
If I remove the autoid/value from the query, it is around 5s, still quite hard for the client to work with when navigating through many documents unfortunately.
If I remove this field, the document-fields endpoint only takes 0.5s.
We’re getting quite concerned as a lot of data entry has taken place and now are worried there isn’t a remedy.
Why do you need the blueprint title here? It means that Kirby has to read all the blueprints on top of the pages, so I would expect this to slow things down.
Looping through the complete index is something you should also avoid on large sites if possible.
There are plugins like Kirby Boost that can help improve performance for such queries on large sites:
I need the blueprint because sometimes there might be two pages with the same title but are different types, so the client needs a way to distinguish. Removing it does speed up the query to about 4s, which is still quite slow however.
I’ve tried installing boost but realize I didn’t configure it properly. I’ll give it a go and see if it helps things.
Thanks a lot @texnixe it already looks like changing to title: {{ page.intendedTemplate }} - {{ page.title }} (sorry should have been in title, not value) already helps a lot - the field only takes about 1s to load now.
@bnomei sorry to rope you in here, but I’m wondering how one could use Boost to speed things up here as the collection query is just filtering by intendedTemplate.
boost will make it faster to access data that is stored inside the content file for boosted templates. so the title and autoid will be loaded from boost cache to the page object and then fill your query. it will be faster.
BUT i think you do not need boost and could might be fine by just adding a cache around the collection and only update the list like once per minute (or flush using hooks).
the boost plugin itself has collections showing what you have now “boostidpage” (just site index with filter) and what you might need “boostidkv” ( a cache with key value data for the multiselect field).
the “boostidkv” is loading the boost index cache (thats a list of all files boostid / diruri / title). what you need would be to create a cache for your collection. you can do that manually or use something like my lapse plugin.
the main difference is that you will not have page.* in the query but arrayItem.*.
return function ($site) {
// I assume all these templates have an autoid to keep lapse fast
$collection = $site->index(true)->filterBy('intendedTemplate', 'in', [
'person',
'event',
'document',
'place'
]);
// ->sortBy('title') // this would be slow
// use lapse to either generate or load data array from cache
$data = Lapse::io(
// use laspe to create a key from collection based on modified values stored in autoid db
['connectable_kv_for_multiselect', $collection],
function () use ($site, $collection) {
// inside callback things are only executed on cache refresh
// its ok to sort here
$collection = $collection->sortBy('title');
// create primitive types of data to serialize
return array_values($collection->toArray(function($page) {
return [
'title' => $page->title(),
'blueprint_title' => $page->blueprint()->title(),
'autoid' => $page->autoid(),
];
});
}
);
// build objects that kirby can use for multiselect field
$kv = array_map(function ($item) {
return new \Kirby\Toolkit\Obj($item);
}, $data);
return $kv;
};
I’m sorry to revive this thread as I haven’t had the chance to try to implement boost and so on, but am encountering an issue now where the document-fields endpoint is not unusably slow but the actual panel view itself is causing issues: https://domain/api/pages/slug?view=panel is taking a staggering 12-14s then tab-fields loads in about 800ms afterwards.
This is on a quad core high performance Intel vCPU Digital Ocean droplet with 8GB RAM and also 13s on my 3.6ghz 8-core i9.
When loading the panel view, I can see one cpu getting hammered to 100% in htop.
@texnixe can you recommend any ideas on how to go about trying to debug this? The client is getting very exasperated as data entry is essentially impossible. The blueprint employs a number of multiselect fields populated with site.find('taxonomies/example').index(true) in case that is particularly slow.
In case this may help with loading issues, I’m trying this lapse solution but am wondering how I can abstract it so I can call this without having to manually build a collection for every single field that needs it.
Is there a way to define the function so that it can be used like:
fetch: customFunction("slug", "key")
–
To answer this particular qusetion, I’ve set this up as a site method that can be called in the multiselect query accordingly:
Then in my multiselect query, I am calling it with fetch: site.getCachedCollection($slug, $key)
I’ve replaced every reference in my blueprint with this call. I can see the plethora of cache files being built in cache/domain/bnomei/lapse, have set up the mysql driver and turned debugging off and the panel is still taking 10s to load (it actually takes almost 40-50s each time the cache is rebuilt).
–
For additional clarity, a number of these multiselect fields are used within structure fields, where the client needs to be able to create multiple annotated associations, say for example in the case of credits:
using a collection or a sitemethod should both work. what is important how the files are cached and used. the problem with fetch in structures is that the panel will load the dataset for each entry again and again to populate the dropdowns.
i had a project like this once and i ended up using blueprints based on php and instead of using a cache file with json i wrote the data to php files and required these in the php-blueprint files.
Interesting that the structure field re-queries each dataset… Is there any information on how to go about creating blueprints in PHP? Do I understand correctly that it would be more optimal as the structure fields would simply be already statically populated on one single render of the panel rather than per row?
writing the blueprints in php is supported but no well documented.
all blueprints use the model::load which used Data::load which is detecting the handler by file extension.
the kirby repo unittests do load from php but mostly inline and not from required files.
yes and no.
yes… it will be a bit faster since the php files of the blueprints and the kv values as php will be compiled into the app (opache, apcu app cache).
no… this is not the an cache of an unrolled blueprint. i created an idea for that but not many seem to run into issues like that (or just dont know where the performance bottleneck is).
(edited: the cache for blueprints will always have problems caching data fetch from external sources. thats nothing kirby can do easily)