Performance issues with ~3,000 virtual pages

The CMS Questions
1

Hi,

we’re currently facing performance issues in the Panel, with a total loading time of around 2000 ms. We have about 100 regular (physical) content pages, which are not a problem.

The bottleneck seems to be the creation of virtual pages. We fetch data from an API and store the raw data in a cache. On the parent page, we override the children() method to return a new Pages object:

public function children(): Pages
{
	if ($this->children instanceof Pages) {
		return $this->children;
	}

	return $this->children = $this->resource->toPages(parent: $this);
}

This generates roughly 3,000 virtual pages. The toPages() method uses Page::factory() to create new Page objects from the cached data, then returns a mapped array of these pages:

public function toPages(Page $parent): Pages
{
	$data = $this->data();
	$pagesData = array_map(
		function ($item) use ($parent) {
			$fields = $this->mapFieldsFromData($this->config->map(), $item);
			$page = [
				...$fields,
				'template' => $this->config['template'],
				'model'    => $this->config['model'],
				'parent'   => $parent,
			];

			return $page;
		},
		$data,
	);
	
	$pages = array_map(
		function ($item) {
			// APIRecord extends Page class
			return APIRecord::factory($item);
		}, $pagesData
	);

	return new Pages($pages, $parent);
}

When we skip the factory calls, loading time drops to about 450 ms, but as far as I know, creating a Pages object requires an array of Page objects, which we can only get by actually creating them for each virtual page — is that correct? We already tried replacing the $fields variable with only the slug and title which only saves around 100ms.

We recently updated to Kirby 5.1 and started using Kirby Turbo, hoping for performance improvements. However, it seems that virtual pages are not cached and therefore don’t benefit from Turbo’s optimizations.

Any tips or insights on improving performance in this scenario would be greatly appreciated.

Thanks in advance!

2

Have you considered not using full Page models for your Panel list?

Instead of returning thousands of Page objects via children(), you could provide the Panel with a custom section that renders plain k-item data (id, text, link, icon, …). That way, you don’t need to instantiate every virtual page just to display them in the Panel, and performance should improve significantly.

3

have you used a tool like SPX to find expensive calls in detail? from my experience the core uuid cache is not well optimized for mass creation of pages. that one thing where my turbo plugin might help in regard to virtual pages as it ships with a different one you can enable. i assume you provide fixed uuids in the content and not let kirby create new ones on the fly.

4

just to clarify: turbo caches page content via their models. it will help to load less files and walk less dirs in file based kirby setups. it can not magically make kirby skip php object (pages factory) creation.

here is what i would do i your case. instead of making them virtual have them created in kirby. every time you refesh the base for the virtual pages remove the old ones first. unless yo all 3000 in every request this will be faster.

or if you want to keep virtual pages then consider to perform filtering on the 3000 on your raw data, before creating the children. that should also speed things up.

5

We enabled the Turbo uuid cache and also provide fixed uuids.

We actually just tried today to save each page to disk by have them created by Kirby, which unfortunately did not help but rather made the performance even worse, doubling our loading time to around 5000 ms. We also tried using Turbo by declaring it in the page’s model but that also did not help. We also ran into memory issues while creating the pages, taking up 1.2GB of RAM during the creation.

We may have to try a custom panel section, like @philippoehrlein suggested, which would be unfortunate since we basically would re-produce the default pages section just with less information loaded at once.

I have never worked with SPX before, will probably look into it, do you have any suggestions to pinning down the issue?

6

Uhm. then there is something broken elsewhere. When you have 3000 content pages, even in a single folder Kirby usually does not break a sweat at all (<100ms) since the panel does pagination for the pages section. So having real pages will certainly be faster than 3000 virtual ones unless…

somehow you force Kirbys Panel to load all 3000 every request, then not.

Maybe there is a missunderstanding of what i meant with creating the pages. I was suggesting to use the json of the 3000 virtual pages and have Kirby create them once and once only. this can be done with the setup you showed above and calling →save() on them. then never fetch the 3000 via the json but rely on kirby default children() implemenation. only flush/recreate them of the base data changed.