How to speed rendering virtual pages imported from CSV?

Hi to all

Faced with the following problem.
Due to the fact that my site uses a geo-detection module and a module that changes content for mobile devices, I can not cache the site. And now there is a task of importing a directory with more than 10000 records from CSV.

Rendering 1000 records takes 2 seconds, and 10000 already more than 20 seconds and it is clear that this is an unacceptable result for the site.

Tell me how I can cache this information so that each time the script did not refer to this table or did it iteratively. I tried different plug-ins, but it did not give a result. Can you tell me which way I should think?

Moreover, initially, I thought to take some information from CSV and edit part of it on the site, but according to the installation written in the examples, it still increases the rendering time of virtual pages x10 times.

I’m hoping very much for help. Maybe there is some way to cache collection via lapse plugin?

As I understand from post Panel speed / cache?
The Bottleneck is here Pages::factory

You can use CSV, SQlite or MySQL but for showing the content I need to create virtual pages and every time I give a task to generate 10000+ rows via Pages::factory
It loads the system and the pages are generated for a very long time.
Maybe there’s something I don’t understand?

Hi, i wrote that post.

I had to carry out a project where I had to read from a section of the panel a sqlite database of 35,000 medical authorizations.

Like you, it took about 30 seconds to load the panel section, even though it was paginated every 10 pages.

1.first use last version of php: with php7.4
2. Order directly from the database and not from the bluprint. I directly remove the part code that allows ordering from the blueprint.
3. i wrote a custom section plugin with some optimizations & features. with cache (best result usign apcu driver), custom query directly from section php code, dont query all columns only query waht u use, sort directly from de query.
4. i use sqlite, because working from file system on such big data the “file I/O” collapse my server


use Exception;
use Kirby\Cms\Pages;
use Kirby\Cms\Site;
use Kirby\Toolkit\A;
use Kirby\Toolkit\Query;

$pagesSection = require kirby()->root('kirby').'/config/sections/pages.php';

return A::merge($pagesSection, [
    'props' => [
        'query' => function($query = null) {
            return $query;
        'search' => function ($search = true) {
            return $search;
    'computed' => [
        'pages' => function () {

            $authCache = kirby()->cache('af');
            $authData = $authCache->get('authdata');

        if($authData === null){

         $authData = [];

        $dba = Db::select('datos', ['fecha','idc','nota_interna', 'observaciones' ,'localidad', 'especialidad', 'profesional','plantilla', 'slug', 'title',  'status', 'nro_afiliado', 'prestador', 'codigos_manuales', 'codigos_nn'], null, 'fecha DESC, slug DESC');

       foreach ($dba as $autorizacion) {

              $authData[] = [
                'slug'     => $autorizacion->slug(),
                'num'      => $autorizacion->status() === 'listed' ? 0 : null,
                'template' => $autorizacion->plantilla(),
                'model'    => $autorizacion->plantilla(),
                'content'  => [
                    'title'  => empty($autorizacion->title()) ? $autorizacion->idc() : $autorizacion->title() ,
                    'status' => is_null($autorizacion->status()) ? 'unlisted' : $autorizacion->status(),
                    'fecha' => $autorizacion->fecha(),
                    'nro_afiliado' => $autorizacion->nro_afiliado(),
                    // 'fecha_emision' => $autorizacion->fecha_emision(),
                    // 'fecha_recepcion' => $autorizacion->fecha_recepcion(),
                    'prestador' => $autorizacion->prestador(),
                    // 'coseguro' => $autorizacion->coseguro(),
                    'codigos_manuales' => $autorizacion->codigos_manuales(),
                    'codigos_nn' => $autorizacion->codigos_nn(),    
                    // 'diagnostico' => $autorizacion->diagnostico(),
                    'observaciones' => $autorizacion->observaciones(),
                    // 'informe' => $autorizacion->informe(),
                    // 'creado' => $autorizacion->creado(),
                    'nota_interna' => $autorizacion->nota_interna(),
                    // 'mensajeria' => $autorizacion->mensajeria(),
                    // 'pdf' => $autorizacion->pdf(),
                    'localidad' => $autorizacion->localidad(),
                    'especialidad' => $autorizacion->especialidad(),
                    'profesional' => $autorizacion->profesional(),
                    // 'diagnostico' => $autorizacion->diagnostico(),
                    // 'memo' => $autorizacion->memo(),


          $authCache->set('authdata', base64_encode(gzcompress(serialize($authData))), 15);
          $authData = base64_encode(gzcompress(serialize($authData)));


        $mo = pages('dbautorizaciones');

        $pages = Pages::factory(unserialize(gzuncompress(base64_decode($authData))), $mo);


            // search
            if ($query = get('q')) {
                if(strpos($query, '+') !== FALSE){
                    $query_arr = explode('+', $query);
                    foreach($query_arr as $query){
                        $pages = $pages->search($query);
                } else {
                     $pages = $pages->search($query);



            // // loop for the best performance
            // foreach ($pages->data as $id => $page) {

            //     // remove all protected pages
            //     if ($page->isReadable() === false) {
            //         unset($pages->data[$id]);
            //         continue;
            //     }

            //     // filter by all set templates
            //     if ($this->templates && in_array($page->intendedTemplate()->name(), $this->templates) === false) {
            //         unset($pages->data[$id]);
            //         continue;
            //     }
            // }

            // // sort
            // if ($this->sortBy) {
            //     $pages = $pages->sortBy(...$pages::sortArgs($this->sortBy));
            // }

            // // flip
            // if ($this->flip === true) {
            //     $pages = $pages->flip();
            // }

            // pagination
            $pages = $pages->paginate([
                'page'  => $this->page,
                'limit' => $this->limit

            return $pages;
        'add' => function () {
            if ($this->query) {
                return false;

            if ($this->create === false) {
                return false;

            if (in_array($this->status, ['draft', 'all']) === false) {
                return false;

            if ($this->isFull() === true) {
                return false;

            return true;
    'toArray' => function () {
        return [
            'data'    => $this->data,
            'errors'  => $this->errors,
            'options' => [
                'add'      => $this->add,
                'empty'    => $this->empty,
                'headline' => $this->headline,
                'help'     => $this->help,
                'layout'   => $this->layout,
                'link'     => $this->link,
                'max'      => $this->max,
                'min'      => $this->min,
                'search'   => $this->search,
                'size'     => $this->size,
                'sortable' => $this->sortable
            'pagination' => $this->pagination,

I’m going to check the code well to see other optimizations made because it was a while ago and I don’t have it in mind. i will write later with more info.


PD: from 30seg (or never load) to 3/4 seconds (working on cheap shared hosting, php7.4) | 4/5 seconds on php7.3

Thanks @diezmilseres for your solution.
I’m not use panel. My collection is used only on frontend as virtual pages.
But anyway, I’ll study the code, and maybe I can speed up the loading of the pages.

Hi again,

im still trying to find variant of speedup loading my games catalouge.

Im create module

class GamesPage extends Page
    public function children()
        $csvCache = kirby()->cache('games');
	    $csv  = $csvCache->get('gamesData');
	    if ($csv === null) {
	      $csv = csv($this->root() . '/games.csv', ';');
	      $csvCache->set('gamesData', base64_encode(gzcompress(serialize($csv))));
        $children = array_map(function ($game) {
            $slug = Str::slug($game['Name']);
            $software = Str::slug($game['Software']);
            return [
                'slug'     => $slug,
                'template' => 'game',
                'model'    => 'game',
                'num'      => 0,
                'content'  => [
                    'title'       		=> $game['Name'],
                    'description'  		=> $game['Description'],
                    'text'  			=> $game['Review'],
                    'iframe_url' 	 	=> $game['Demo URL'],
                    'mobile_url'  		=> $game['Mobile URL'],
                    'date' 				=> $game['Release Date'],
                    'gaming_software'  	=> $game['Software'],
                    'cover'  			=> $slug.'-'.$software.'-logo.jpg',
                    'background'  		=> $slug.'-'.$software.'-bg.jpg',
                    'rating'  			=> $game['Rating'],
                    'rtp'  				=> $game['RTP'],
                    'volatility'  		=> $game['Volatility'],
                    'reels'  			=> $game['Reels'],
                    'paylines'  		=> $game['Paylines'],
                    'min_bet'  			=> $game['Min Bet'],
                    'max_bet'  			=> $game['Max Bet'],
                    'games_category'  	=> $game['Category'],
                    'games_type'  		=> $game['Type'],
                    'games_features'  	=> $game['Features'],
                    'games_themes'  	=> $game['Themes'],
        }, unserialize(gzuncompress(base64_decode($csv))));
        return Pages::factory($children, $this);

And even set cache for it, but as I understad main problem is here return Pages::factory($children, $this);

So, It is better to cache data in the controller?

I was trying to cache
$games = $page->children()->listed()->sortBy('date')->flip();

but this isn’t work

$gamesChildren = kirby()->cache('games');
    $games  = $gamesChildren->get('gamesChildren');

    if ($games === null) {
      $games = $page->children()->listed()->sortBy('date')->flip();
      $gamesChildren->set('gamesChildren', base64_encode(gzcompress(serialize($games))));

it wrote error: Call to a member function paginate() on string
for line: $games = $games->paginate($limit);

Or this isn’t help? and no other way? @texnixe maybe you can help with this?

Can you share de csv to test it?

I’m really sorry to be not more helpful about the topic, but this doesn’t seem like a nice thing to do! Content on a URL should be the same for each device. AFAIK search engines even punish you if you use this pattern (and it’s very annoying for users who suddenly can’t find information any longer when they’ve switched devices).

To get back into the topic: maybe if you fix this pattern, you can actually cache it and your performance issue will be dealt with equally.

1 Like

I mean that this changes are change design and some elements like filters. I know that I can do all by css, but for some elements its better to use separeted styles.

I think I understand what I need to do.

The problem is here:
In controller $games = $page->children()->listed()->sortBy('date')->flip();
im asking for get all info about pages.
If I slice it to 200 it works super fast.

Maybe I can get from this only amount of pages and keys like slug
and then create a limited collection that will have all info just for current page ?
and when I click on next page get next collection based on keys?

I found the reason that is slowdown page loading
I have filters in the left sidebar that count amount of games filtered by some field (software, theme, features etc)

So, every time the page is loading this script is calculating every item.
when you select some software for example script is hide all other providers but still showing other fields. and items recalculated again.

Im use this (<?= $cgames->filterBy('games_category', $category, ',')->count() ?>)
for counting every taxonomy
so maybe it is better to do it with filter() or find? for more quick count?
after disabling this scrtips, speed of loading page was reduced from 12 to 2 seconds on 6500 games catalouge.