Handling external API Data in Kirby - Best Practices & Troubleshooting

The CMS Questions
1

Hi everyone, I’m new to working with APIs in Kirby and have run into a few issues.

What I’m trying to achieve:

  • Make an API call to an external service
  • Reassemble the necessary data
  • Display the data as cards inside a component/snippet wherever it’s used on the website. The cards will not have any further interaction. Just displaying the data.
  • Side info: the data will update daily. So it will have to make an API call once a day.

I followed this guide: Content from an API.

My questions:

  1. Based on the guide, I would create a page model and save the API data as children of a Page. But is this the best approach for my component? I’m not sure if this logic should go inside the config, a hook, or somewhere else. Using children feels a bit odd, but maybe that’s the way to go?
  2. I’ve successfully fetched and structured my data, but I’m getting confused between Kirby objects and standard arrays. What’s the best way to handle JSON data in Kirby? What format should I use?

Here’s some code:

Making the API call and structuring the data:

$response = Remote::request($api, [
    'method' => 'GET',
    'headers' => [
        'X-API-Key' . $password,
        'Authorization: Basic ' . base64_encode("$username:$password")
    ],
]);;

if ($response->code() === 200) {
    $data = $response->json(true);
}

$result = [];

foreach ($data['rows']['cards'] as $card) {
    $version = $card['items']['version']['title'] ?? null;
    if ($version) {
        $result[$version]['cards']["en"][] = [
            'id' => $card["id"],
            'title' => $card['item']['en']['title'],
            'text' => $card['item']['en']['desc'],
        ];
        $result[$version]['cards']["fr"][] = [
            'id' => $card["id"],
            'title' => $card['item']['fr']['title'],
            'text' => $card['item']['fr']['desc'],
        ];
    }
}

foreach ($result as $key => $data) {
    $pages[] = [
        'template' => 'cards',
        'slug' => "version-" . $key,
        'model' => 'cards',
        'content' => [
            'version' => $key,
            'data' => $data,
            'uuid' => Uuid::generate(),
        ]
    ];
}

Inside the snippet:

when I print the data inside my snipped it gives me this format:

Kirby\Content\Field Object
(
    [data] => Array
        (
            [cards] => Array
                (
                    [fr] => Array
                        (
                            [0] => Array
                                (
                                    [id] => 154
                                    [title] => Lorem Ipsum
                                    [text] => Lorem Ipsum
                                )

                        )

                    [en] => Array
                        (
                            [0] => Array
                                (
                                    [id] => 154
                                    [title] => Lorem Ipsum
                                    [text] => Lorem Ipsum
                                )

                        )

                )

        )

)

How do I print this? :sweat_smile: Nothing I tried worked… for example: $item->data()["cards"]["fr"]

Help is much appreciated! :slight_smile:

2

As I understand it, $item is a Page object, and $item->data() is a magic method which returns a Field object for the data content field of that page. A more explicit way to access the same Field object (which I personally prefer because it results in better typing in an IDE with good PHP support) would be: $item->content()->get('data').

Now the question is: when you created the page, what did the content of the data field look like? You’re showing some code where you create some PHP arrays to describe your page, but you’re not showing how you create Page objects from that, and you’re not showing whether you’re saving those Page objects to disk and/or if you’re serializing the PHP value for the data field.

What does the raw string content of $item->data() look like? You can inspect it with var_dump($item->data()->value) (or var_dump($item->content()->get('data')->value)).

In Kirby, page fields are always text values. If you want a field to contain some structured data, that data should be serialized into some format: Markdown, XML, JSON, YAML, etc. If you’re not explicitly serializing the data, maybe your data field ends up containing PHP’s default text representation for an array, basically the same thing PHP would print when using var_dump($data). That’s not great, because it’s not exactly a standard format, and I don’t think Kirby offers methods to parse it as in-memory structured data.

So if you’re creating Page objects and saving them as pages on disk, you probably need a data flow that looks like:

  1. Make the request and get raw JSON.

  2. Parse the raw JSON into a PHP array with the $response->json() method.

  3. Create the data structure for your page(s). For fields which should contain structured content (and not just an unstructured string), convert that PHP array content to a YAML or JSON string (helper classes are available: Json | Kirby CMS Yaml | Kirby CMS).

  4. Save the page to disk.

  5. Then when rendering the page, you can use the Field object’s methods to parse the string data from YAML or JSON back into a PHP array:

// to decode YAML content in a 'data' field:
$page->content()->get('data')->yaml();

// to decode JSON content in a 'data' field:
\Kirby\Data\Json::decode(
  $page->content()->get('data')->value
);

(I thought there’d be a field method to decode JSON, but apparently not.)

As a side note, if you’re storing data as YAML in a page field, you may want to look at the Structure field type which could let you show this content in a readable way in the Panel, and the $field->toStructure() method instead of $field->yaml() when reading that data in your models/controllers/templates.

3

There are different approaches for that.

  • You can create a CRON job that runs once a day and runs a custom PHP script.
  • You can rely on visits to the site and use Kirby’s rendering logic to check things like “do I have a specific page already?” and “what’s the latest update timestamp for that page?” (in that case I’d create a page field on that page with a date), and then if the last update is more than a day old (or 12 hours old or some other criteria) it would trigger the logic that calls that API and recreates/updates your page(s) where you store the latest content from that API. (That means that, every day, the first visit the requests data you’re mirroring from this external API will be responsible for updating said data.)

The first one is easier but can be a bit finicky. The second one is more complex because you have to create a custom mechanism for caching data and invalidating said cache.

As a side note, it might be worth considering where the best place to store your copy of the API data is:

  1. Inside pages that you create programmatically in the content folder?
  2. In a custom cache?
  3. In a database?

If you have this content in a database or in a custom cache, you can still treat it as a set of pages with virtual pages.

4

Good morning,

Thank you for your detailed reply, fvsch! I’m not sure if I fully understood everything, but I’ll read up on it :sweat_smile:.

After a good night’s sleep and a fresh mind, I realized that my current approach doesn’t really make sense… The page model, as the name suggests, works on a page level. It fetches data whenever the defined page template is used. However, what I actually need is a way to store data globally so I can access and display it inside a component that might be used across different page types.

If I understand correctly, one possible approach could be:

  1. Fetching the data from the API as soon as someone visits the website. For example:
'routes' => [
    [
        'pattern' => '(:any)',
        'action'  => function () {
            return // make API call and store it as a text file; But also check if the data has been updated already
        }
    ],
]

Or, I suppose this is where the cron job could come in—setting up a designated route that gets triggered to fetch and store the data :thinking:.

  1. Storing the data in a text file, the same way how Kirby structures its data.
  2. Displaying the content as usual within the templates.
5

setting up a designated route that gets triggered to fetch and store the data

That’s a DDoS risk. HTTP requests to a resource should not trigger expensive computation if it can be avoided, and especially not on every request without caching.

You can have a PHP script (or a script written in any language you like) that is not accessible to the outside world (i.e. not accessible on the website at all) and gets triggered by the cron job. If you’re more familiar with PHP and want to use the utility classes and functions provided by Kirby, it should be possible to create a script that makes use of Kirby code. Basically use the same PHP setup code as the public index.php, but don’t call $kirby->render() (which handles the routing, finding content, executing controllers and templates, and preparing a HTTP response), but instead do your own HTTP requests and optionally work with Kirby classes like Kirby\Cms\Page to construct pages and save them to disk.

6

By the way, if this looks complex, that’s normal.

It’s not that complex, but it’s a full-stack web development task which combines ~5 layers of concern, each with possible solutions:

  1. “When does work happen?” (Cron job, or rely on user requests, or some other solution)
  2. “How do I get data from an external API?” (HTTP requests, authentication)
  3. “How do I persist this data?” (Writing to disk, as content pages or cache content or something else.)
  4. “When visitors request this data, how do I retrieve it from its persisted state?” (Kirby content pages, or virtual pages, or custom routes with arbitrary data read from a JSON file, etc.)
  5. “Now that I have data ready to serve, how do I render it?” (Templating, styling.)

Specialist developers (e.g. backend dev, frontend dev) will often know only a few of those layers. Junior developers may not have a full picture of the layers involved, and often started learning with layers 4–5, and may not be experienced with layers 1–3, because a lot of training material is on the more visible stuff that gets you a result on screen quicker.

If you feel like you’re a bit out of your depth, take the time to learn a bit about each layer and find a solution for each, and/or get help from a collaborator on some of the layers you’re struggling with.

7

ok, i found a solution which works nice I think.
I make an api call inside a contollers/site.php and there I store the data inside the cache (which I have set up). If the cache contains some data → it uses this data, if the cache has expired → it makes another api call.

Thanks for the input :+1:

8

By the way, it looks like this recently announced plugin could have been a decent all-in-one solution for your use case: