Kirby 4 Huge Collection Question

1

Hi,

What is the default order the below collection fetch would use, or would it be just random each time?

$articles = page('articles');
$articlesPageChildren = $moviePage->children();

and then i the loop:
$pages = $articlesPageChildren->slice($ndx * $limit, $limit);

If I have 1 milion articles pages that i want to process and that process fail at 200k, if i execute the above again would they be fetched in the same order?

When i try to use $moviePage->children()->sortBy(‘sort’, ‘asc’) i think it tries to fetch all at once and it takes forever somehow.

Any advice?

2

Is it really about 1,000,000 articles? Basically, I would use a strategy for a high number of pages and group the content into folders. For example, for a movie collection according to this structure:

content/
└── movies/
    ├── 2023/
    │   ├── a/
    │   │   ├── amazing-movie/
    │   │   └── another-great-film/
    │   ├── b/
    │   │   └── brave-adventure/
    │   ├── c/
    │   │   ├── cool-story/
    │   │   └── captivating-drama/
    ├── 2024/
    │   ├── a/
    │   │   ├── awesome-film/
    │   │   └── ancient-legend/
    │   ├── m/
    │   │   └── mighty-quest/
    │   ├── s/
    │   │   ├── silent-night/
    │   │   └── summer-breeze/
    │   └── other/
    │       └── zeta-endgame/
    └── genres/
        ├── action/
        │   ├── action-movie-1/
        │   └── action-movie-2/
        ├── comedy/
        │   ├── funny-movie/
        │   └── laugh-out-loud/
        └── drama/
            ├── emotional-journey/
            └── heartfelt-story/

Alternatively, you can also work with a database:
https://getkirby.com/docs/guide/virtual-content/content-from-database
https://getkirby.com/docs/guide/database

3

1 milion was an example, i see around 500K, data is being stored in redis which builds collection of pages out of it.

Any idea about the sort questions?

4

Default sort order is by order in the file system if you do not sort by anything else.

5

How efficient is sortBy(‘sort’, ‘asc’), is it loading all pages into memory? I am guessing if i do sortBy with limit and offset it will load everything into memory as well, the whole collection of pages?

Which in my case will run out of memory.