Virtual page model slows down site with many pages

The CMS Questions
1

Hello!

I’m having performance issues with one of my Kirby websites.

For this project, my goal was to fetch project-related events via an API and combine them with content from Kirby CMS, such as images and additional event information.

So basically I built myself a custom page model plugin that uses Merging content sources | Kirby CMS as a basis for a content merge.

# more code 

public function children(): Pages
{
    if ($this->isUnlisted()) {
        // unlisted = archived projects, ignore those
        return Pages::factory([], $this);
    }

    if ($this->children instanceof Pages) {
        return $this->children;
    }

    $pages = new Pages();

    $arrangementId = $this->arrangementId()->value();
    $results = site()->getProjectAllArrangementEvents()[$arrangementId] ?? [];

    $parameters = array_column(
        site()->getProjectAllOptions(),
        'value'
    );

    if ($results) {
        foreach ($results as $event) {

            // only fetch events that are public!
            if (
                !is_array($event) ||
                ($event['public_status'] ?? null) != 1 ||
                empty($event['event_id'])
            ) {
                continue;
            }

            // Get the page on disk, if it exists
            $slug = Str::slug(
                $this->arrangementId() . '_' . $event['event_id']
            );

            $page = $this->subpages()->find($slug);

            $content = [];

            foreach ($parameters as $parameter) {
                if (isset($event[$parameter])) {
                    // Prefix Project fields to avoid conflicts with Kirby fields
                    $content['project_' . $parameter] = $event[$parameter];
                }
            }

            // Custom Kirby fields

            $content['kirby_more_event_info'] =
                $this->kirbySubtitle()->value() ?: '';

            $content['kirby_additional_info'] =
                $this->kirbyAdditionalInfo()->value() ?: '';

            // ... more fields ...

            $virtualPage = Page::factory([
                'slug'     => $slug,
                'template' => 'calendar_event_project',
                'model'    => 'calendar_event_project',
                'parent'   => $this,
            ]);

            try {
                $virtualPage->changeStorage(MixedStorage::class);

                $virtualContent = $content;

                $virtualContent['uuid'] =
                    $page?->uuid()->toString() ?? Uuid::generate();

                $virtualContent['kirby_panel_url'] =
                    $virtualPage->panel()->url();

                $languageCode =
                    kirby()->language()?->code() ?? 'default';

                $virtualPage->storage()->writeVirtual(
                    versionId: VersionId::latest(),
                    language: Language::ensure($languageCode),
                    data: $virtualContent
                );

                $pages->add($virtualPage);

            } catch (Exception $e) {
                error_log(
                    'Failed to create virtual page for event ' .
                    $event['event_id'] . ': ' .
                    $e->getMessage()
                );
            }
        }
    }

    return $this->children = $pages;
}

Info:
in getProjectAllArrangementEvents() I cache the response which improved the performance a lot. But the php processing is still intense :face_with_crossed_out_eyes:

My content structure looks like this:

01_project
  |_event_01
  |_event_02
  |_event_03
...
  |_event_99_or_more

When I first built this, the number of pages using this page model was much smaller. By now, there are roughly 100 times as many pages, and I’ve noticed that the website has become significantly slower as a result.

so my question is:
Does anyone have a hint on how to make this faster … or what would be a way to prevent this kind of performance lag when working with a large number of virtual pages?

Or, if you have an idea for a better approach than using virtual pages in this case, I’d be very interested to hear it :sun_with_face:
Thanks a lot!

2

it seems to me you are calling a lot of outbound methods in the nested foreach loops. and that adds up, things like $this->subpages()->find($slug);.

you could try my GitHub - bnomei/kirby-api-pages: Virtual Pages from APIs · GitHub plugin and see if its faster.

3

Do you perhaps know a more efficient/resource friendly way to do this without the find() method?
:thinking:

I’ll definitely check out your plugin as well!

4

my guess is that doing it manually up front in using Dir::index to get all folders, same for the drafts folder, strip X_-prefix and exclude slugs from an array and that way might be faster than repeated find calls. adding all items with a single append call might also be faster than repeated add to collection.

but i would sprinkle some mircosecond timers in between the calls to first measure where most time is spent.

5

Hey bnomei

Since I load the subpages before, I used them to generate me an array with slugs

    protected $subpages = null;

    public function subpages()
    {
        return $this->subpages ??= Pages::factory($this->inventory()['children'], $this);
    }

and then I replaced the find() method with

  $page = $subpagesBySlug[$slug] ?? null;
  // $page = $this->subpages()->find($slug);

this is roughly one quarter faster than using the find() method.

Thank you for the hint :slight_smile:

6

Hey :slight_smile:

Yesterday I got a little deeper into the websites logic and I found another performance issue:

When I load pages where no page cache has been written yet, the virtual page logic above is sometimes triggered, even though none of that logic is involved in those templates.

For example: when I’m on the jobs page, all the virtual pages related to the event calendar are still being processed which results in a (unnecessary) lag. Then the page cache is written and
every thing is good :+1:

I found this out when I looked at my error logs. There, I could see that visiting those pages triggers the timing logs I used to debug the performance issues here.

It may come from little helpers I created to reference to specific pages more easily:

# inside my siteHelper Plugins index.php
# I've added siteMethods to reference to special pages like this:

'getJobsPage' => function () {
    return site()->children()->children()->filterBy('intendedTemplate', 'jobs')?->first();
},

I use this for example In a block module, that is rendered inside the jobs page template:


<?php if(!isset($hasPadding)) {$hasPadding = false;} ?>
<?php $availableJobs = $site->getJobsPage() ? $site->getJobsPage()->children() : new Pages() ?>
<?php if($availableJobs->count() > 0): ?>

# html here

<?php endif ?>

I’m wondering if there’s a better way to reference those pages.

I clear the cache every day, so the client has noticed some slowdowns, mainly on the less frequently visited English version of the website.

Maybe you can help me with this one as well :slight_smile: