Batch-uploading .txt files

Kirby 3 Questions 🤔
1

Hi all,

I’m transferring a(n already existing) small library catalog (~ 2.500 books) to Kirby. After using some awkward CSV export/Excel scripting, I managed to export an individual .txt file for each book that corresponds with the blueprint I’ve set up in Kirby.

This is one example:

Title: Dubuffet Typographer

----

Author: Leguillon, Pierre

----

Abstract: A tribute to the French artist Jean Dubuffet (19011985) and to the typographic strategies with which Dubuffet achieves, on a visual level, his plan to destroy language through books and lithographs. Traveling to public and private archives such as la Fondation Dubuffet in Paris, the Bibliothèque Kandinsky at Centre Pompidou or to IMEC in Caen, Pierre Leguillon has photographed ephemera such as invitations, posters, catalogs, artist's books, flyers, tickets, and record sleeves. These images were then used to execute a recadrage (re-framing) of Jean Dubuffet and his activities. Like a meticulous detective, Leguillon shows us how for each project, Dubuffet invented a new way of writing and composing text quite possibly by simply improvising.

----

Pages: 364

----

Year: 2014

----

Publisher: Presses Du Reel

----

City: Dijon

----

Url: https://www.lespressesdureel.com/EN/ouvrage.php?id=2960

----

Isbn: 9782930667058

----

Number: 2670

----

Category: Typography

----

Location: Arnhem

----

Status: available

Now I’m wondering: What would be the smartest way to batch-upload and structure these ~2.500 .txt files? It would be fantastic to avoid creating subfolders as they are not really necessary in this case.

Also, the files are currently named after the respective ‘book number’ (which is just incremental, so they go from 1 all the way through ~2.500).

Curious to hear how this could be done!

Many thanks,
Leon

2

What does that mean? Not sure I understand.

If I get you right, you want to find a way to loop through all of these text files, create a folder for it that is named after what and then remove the number from the text file?

3

When you want to use each book as a page (or object) in kirby, you have to encapsulate each txt file in its own subdirectory…

But, if you already have all of them in a CSV, and you don’t want to manage them in Kirby (only reference them), you could use virtual pages for that.

4

It would probably be less work to start over again and directly create Kirby pages from the .csv data instead of textfiles that have then to be moved into Kirby folders and renamed.

5

Thanks for your reply! My question was not so much about creating a folder for each file, but rather if there would be way to avoid using subfolders at all (i.e. having all the .txt files in one directory).

6

Thanks, I’ll look into that! Do I understand it correctly though that a) there is no way to bypass subdirectories and b) using virtual pages would not allow me – or other users – to edit anything directly in Kirby (so any changes would have to be made in the referenced file itself, e.g. the CSV document)?

7

I see, that makes sense. Thank you!

8

If you use a database, you would be able to edit this from the Panel. However, a .csv file cannot easily be updated, so I’d say, no.

Folders for pages are mandatory. There is no way whatsoever to only use textfiles without putting them into folders. It wouldn’t work, because your text files to work with blueprint would have to use the name of the blueprint, not file names with numbers in it.

9

Thanks so much for clarifying this! I’m afraid the editability within Kirby is kind of crucial in this project. Is there a best-practice way to generate pages from a .csv file (without directly referencing it)?

10

You can use the function from the Virtual pages guide to read your csv data, then use the createChild() method to create pages from it, either $site->createChild() or $page->createChild() depending on whether those pages should be direct children of the site, or subpages of a parent page:

1 Like
11

Sounds great, thanks for this!

12

I went through the Virtual pages guide step by step, but I’m not sure where exactly to use $site->createChild()?

For context, this is how my books model looks like now:

<?php

class BooksPage extends Page
{

    public function children()
    {
        $csv      = csv($this->root() . '/books.csv', ';');
        $children = array_map(function ($book) {
            return [
                'slug'     => Str::slug($book['Number']),
                'template' => 'book',
                'model'    => 'book',
                'num'      => 0,
                'content'  => [
                    'title' => $book['Title'],
                    'author' => $book['Author'],
                    'abstract' => $book['Abstract'],
                    'designer' => $book['Designer'],
                    'language' => $book['Language'],
                    'pages' => $book['Pages'],
                    'year' => $book['Year'],
                    'publisher' => $book['Publisher'],
                    'city' => $book['City'],
                    'url' => $book['Url'],
                    'isbn' => $book['Isbn'],
                    'number' => $book['Number'],
                    'category' => $book['Category'],
                    'location' => $book['Location'],
                    'status' => $book['Status'],
                ]
            ];
        }, $csv);

        return Pages::factory($children, $this);
    }

}

Or does this only work for virtual pages?

13

That’s not the right way to go, the model is for creating Virtual Pages. I only pointed you to the Virtual pages guide because of the csv function.

So nothing but this line:

 $csv      = csv($this->root() . '/books.csv', ';');

is relevant.
The rest would be looping through this array and call the createChild() method.
You can put this your code into a template or use a route to actually create the pages.

14

I’m afraid I’m a little stuck on that. So the necessary code would need to look like this?

foreach($csv = csv($this->root() . '/books.csv', ';');)->createChild()

Also, just to understand this correctly: How does Kirby know which columns from the CSV file correspond to the fields set up in the blueprint?

Sorry for the confusion!

15

Nope.

 $csv      = csv($kirby->root('index') . '/books.csv', ';'); // correct path to the file
 // do a dump, so you see what you have, make sure to use the correct delimiter
dump($csv);
 foreach ($csv as $book ) {
  $newBook = $site->createChild([
    'slug' => $book['whatever'], // something unique here
    'template' => 'some-template',
    'content' => [
      'title' => $book['whatever-is-supposed-to-be-the-title'],
      // other key => value pairs for content
    ]
]);

// if you want to make the page listed, you can publish it now
$newBook->changeStatus('listed');

Note that if you are not logged in to the Panel, you need to authenticate before running this code

$kirby->impersonate('kirby');
16

Thanks so much for this detailed answer! I think I’m getting closer to the desired result. However, when I run the code, I get the error message

array_combine(): Both parameters should have an equal number of elements

pointing to this line

$a = array_combine($csv[0], $a);

from the index.php file of the csv function:

<?php
function csv(string $file, string $delimiter = ';'): array
{
    $lines = file($file);

    $lines[0] = str_replace("\xEF\xBB\xBF", '', $lines[0]);

    $csv = array_map(function($d) use($delimiter) {
        return str_getcsv($d, $delimiter);
    }, $lines);

    array_walk($csv, function(&$a) use ($csv) {
       $a = array_combine($csv[0], $a);
    });

    array_shift($csv);

    return $csv;
}

Could it be that this issue is related so something within the CSV file itself, e.g. line breaks?

17

Is the delimiter correct and your csv file well formed, i.e. has the same number of field in each line?

18

I set the delimiter to ; (in the csv file, the book.php template and the index.php of the csv function). All rows in the csv have the same amount of columns, however some fields in the table are empty (they’re still separated by a ; though).

19

Hm, but array combine complains because of a different number of elements in the first line of file $csv[0] (the line that contains your fields) and one of the data rows, which can easily happen if a field contains the separator as well, I think.

20

A few hours of head-scratching later, I finally got it to work. Two things were crucial in that process: Cleaning up the CSV file (semi-colons, leading or trailing whitespaces, line breaks etc.) and extending the maximum execution time for the script itself. Might be obvious, but wasn’t to me, so I thought I’d just post this here for further reference. Thanks to everyone for their help, I appreciate it!