Breaking content directory down by year/month to improve performance

[geodude]

Hi there,

My site has a content directory with almost 7000 entries in it. I want to improve the performance when retrieving content from said directory. I’ve read elsewhere on this forum that splitting a large directory in to chunks by date (month/year) can help to reduce server response times.

See here : Server response time - #7 by Svnt

Can anyone advise how I would achieve and directory layout of of directory > years > months?

Here’s a screenshot of how my directory currently looks.

Any help would be greatly appreciated.

[texnixe]

What exactly do you want to know? How to move them programmatically? How to set up the blueprints? Other stuff?

[geodude]

I want to know how to create sub directories programmatically, based on a date field. Also, I would need the shows I’ve already created, as well as future shows to sort themselves.

I’ve had a read through the docs, forum etc and I don’t know where to start.

[texnixe]

I’d start with creating the blueprints for the years and month pages, where you also set the numbering scheme for these files.

Then you can first pluck all the years from the existing pages and create the year folders from this years array.

Do the same for the month of each year.

Then move the articles into the corresponding folders.

You seem to use alphabetical numbering, so you wouldn’t have to change anything.

[geodude]

Okay, do you have guides on how to create year templates and what defs I need to give them?

[texnixe]

Basically, your years list the months as subpages and the months the articles for each month, so you can basically just use the pages preset for the blueprints: https://getkirby.com/docs/reference/panel/presets/pages if you don’t want anything specific.

Or create those with columns and section for drafts, listed and unlisted. All depends on your needs.

The docs on blueprints and the examples should help you.

Introduction

Samples

[geodude]

Okay thanks. I guess I was really looking for how I would filter the ‘Shows’ so only the ones from each month would be attached to say ‘February’ is that done automatically based on the folder structure or is there a function I need to add to the blueprint definition.

[texnixe]

No, you would actually have to move your show seither manually (too much work) and therefore programmatically into their respective target folders. There is no magical way to achieve that via a blueprint setting.

Dir::move()

[geodude]

Hi @texnixe,

I’ve taken your advice and built a script which sorts all my pre-existing shows into the correct content folders by year > month. See below for a screen shot of how this looks…

I’m now facing another problem where if I would like to grab all shows at once, I don’t know how this could be done.

The best I’ve got so far is this…

$bolt = bolt('shows')->children()->find('20' . date("y"))->find('12', '11', '10', '09', '08', '07', '06', '05', '04', '03', '02', '01')->children()->listed()->filterBy('featuredMix', true)->sortBy('date', 'desc');

This works well for searching the whole of the current year… but what If I want to search back through multiple years?

Is there a function which would allow me to grab all show.txt’s from the shows directory? Essentially bypassing the fact they are sorted in to sub directories.

[texnixe]

Without your bolt function, it would be page('show')->index() and then filter by template. Maybe index also works with bolt.

[geodude]

Awesome. I’ll give that a try.

With the subdirectories in place the URLs for each show page has now changed to include the year and month folders . e.g

http://localhost:3000/shows/2020/09/silver-waves-15th-september-20

I’d like this to just read

http://localhost:3000/shows/silver-waves-15th-september-20

I tried changing this via the routes and I only go to far.

 'routes' => [
        [
            'pattern' => 'shows/(:num)/(:num)/(:all).json',
            'action'  => function ($year, $month, $all) {
                // var_dump($year, $month);
                go('http://localhost:3000/shows/' . $all . '.json');
            }
        ],
    ]

This returns the correct page but doesn’t actually render any JSON data, instead it renders the base 404 page.

Do I need to do something else within the action callback to render the JSON data?

[texnixe]

You don’t want to redirect but return the page.

[geodude]

Could you please elaborate a little more on how I might do this. Feeling a little stumped!

[texnixe]

 'action'  => function ( $year, $month, $all ) {
    if ( $page = page($year . '/' . $month . '/' . $all) ) {
       return $page->render( [], 'application/json' );
    }
  }

Note that you needuse a second route that works the other way round and finds the page in the folders.

[geodude]

[
            'pattern' => 'shows/(:num)/(:num)/(:all).json',
            'action'  => function ($year, $month, $all) {
                // var_dump('shows/' . $year . '/' . $month . '/' . $all);
                $page = page('shows')->children()->children()->children()->find('shows/' . $year . '/' . $month . '/' . $all);

                return $page->render([], 'json');
            },
            'pattern' => 'shows/(:any).json',
            'action'  => function ($any) {



                if ($page = page($any)) {
                    return $page->render([], 'application/json');
                }
            }
        ],

I’ve added this and it returns the JSON fine, but I’m still unsure how to change the URL? I need the front end of my site to use a URL that reads /shows/showname e.g without the year and month.

[texnixe]

One part of this is to actually override the url method in a page model, so that the urls with the year and months don’t appear at all (example can be found here on the forum).

The second part is a second route, you can find multiple example here if you ask search.

[Adspectus]

Do you mean, if your visitor types in the URL as /shows/showname-september-20, she should be redirected to /shows/2020/09/showname-september-20? Then, have a look on the rewrite engine of your webserver, i.e. this would do the trick for the example above and all other shows from september 2020:

RewriteRule "^/shows/([^/]+)-september-(\d{2})$" "/shows/20$2/09/$1-september-$2"

Some caveats: If the year does not always follow the pattern YY -> 20YY you have to refine the rule. And you will need at least 12 such rules for every month or use a RewriteMap statement with your own logic. Furthermore it might be necessary to finetune with flags.

[texnixe]

Well, in this case we don’t want to redirect from /shows/showname-september-20 to o /shows/2020/09/showname-september-20, but redirect from /shows/2020/09/showname-september-20 to /shows/showname-september-20 and when /shows/showname-september-20 is called, we want to find that page in one of the year/month folders and return it.

In Kirby this is achieved with two routes, like here in the old Kirby 2 docs: http://k2.getkirby.com/docs/developer-guide/advanced/routing#omitting-the-blog-folder-in-urls

I don’t think we can achieve this with .htaccess rules.

[geodude]

The problem with this approach is that I end up in a infinite redirection loop and the following error displays.

noods-backend.test redirected you too many times.

ERR_TOO_MANY_REDIRECTS

This my code, adapted to suit my folder structure.

'pattern' => '(:any)',
            'action'  => function ($uid) {

                $page = page($uid);

                if (!$page) $page = page('shows/(:num)/(:num)/' . $uid);
                if (!$page) $page = site()->errorPage();

                // var_dump($page);
                // exit();


                return site()->visit($page);
            },
            'pattern' => 'shows/(:any)',
            'action'  => function ($uid) {
                go('shows/' . $uid);
            }

Any ideas why I might be getting a redirection loop?

[geodude]

I’ve also added this is Models > show.php

<?php

use Kirby\Cms\Page;

class ShowPage extends Page
{
    public function url($options = null): string
    {
        return '/shows/' . $this->uid();
    }
}