Creating content from .csv

Kirby 3 Solved ✅
1

Hi all, I am trying to import content from a .csv file, this thread was very useful so thanks to everyone’s input there.

The trouble is I am getting a “Fatal error: Allowed memory size of bytes exhausted” error. I’ve set my “memory_limit” to 512mb and I guess I could increase this further but I wanted to ask first whether there was anything my code that could be improved?

  // the `csv` function can live in a plugin file or in the template where you want to create the children if its only a one time thing
  function csv(string $file, string $delimiter = ','): array
  {
    $lines = file($file);

    $lines[0] = str_replace("\xEF\xBB\xBF", '', $lines[0]);

    $csv = array_map(function($d) use($delimiter) {
        return str_getcsv($d, $delimiter);
    }, $lines);

    array_walk($csv, function(&$a) use ($csv) {
        $a = array_combine($csv[0], $a);
    });

    array_shift($csv);

    return $csv;
  }

  // Adapt this to your csv file name and the fields in your csv file:
  // Example based on the guide example
  $csv = csv($page->root() . '/test.csv', ',');
  $kirby->impersonate('kirby');
  $i = 0;
  foreach ($csv as $issue) {
    $newPage = page('journal')->createChild([
      'slug' => Str::slug($issue['Name']),
      'template' => 'issue',
      'model'    => 'issue',
      'content'  => [
        'title' => $issue['Name'],
        'date' => $issue['Date'],
      ]
    ])->changeStatus("listed", $i++);;
  };

I started getting the error when I added the “changeStatus” bit at the end. There are 483 entries, which doesn’t seem like a huge amount. I was planning to make this a bit more complicated by adding images into fields as well so I’m concerned I am getting this error already.

Is there anything I can do aside from increasing my memory limit?

2

maybe the double ;; causes an issue?

3

Thanks for the input @hansipete but I think that was just an error when I was pasting the code in.

I changed this to use a database instead of the CSV and tried chunking the table rows but it didn’t help.

$allIssues = Db::select('issues');

$kirby->impersonate('kirby');

$batches = $allIssues->chunk(100);

foreach ($batches as $batch) {

  foreach ($batch as $issue) {

    $newIssue = page('journal')->createChild([
      'slug' => Str::slug($issue->Name()),
      'template' => 'issue',
      'model'    => 'issue',
      'content'  => [
        'title' => $issue->Name(),
        'date' => $issue->Date(),
      ]
    ]);

    $newIssue->changeStatus("listed");

  };

};

Does anyone know a good solution to this problem?

4

I’d apply the sorting number at page creation instead of adding another task with changeStatus().

If the issue continues, create a script to run on the command line.

5

Thanks @pixelijn, I tried this approach and it seems to be working.

6

I am using CSV to Kirby Pages as well, and I am having the extra step of “changestatus” as well.

Mind me asking, how to set the page status and sorting number within the

  • createChild()
  • $page->create()
    methods in one go.

And how about the sorting number, if the sorting number has been changed to another format?

7 
kirby()->impersonate('kirby');
foreach( range(1, 5) as $key) {
  $newPage = Page::create(
    [
      'parent'   => page('blog'),
      'slug'     => 'page-' . $key,
      'template' => 'post',
      'status'   => 'listed',
      'isDraft'  => false,
      'num'      => $key,
    ]
  );
}