Content loss on pages with virtual data

[grommasdietz]

Hi there

I’m working on a custom version of the Shopify plugin, not only migrating to v5, but also extending some functionality, like adding logic for the cart via Shopify Storefront API, add native buttons to sync manually, etc.

While most of the migration works fine, I run into a persistent issue with saving content on the product page that I haven’t been able to solve. The products mix virtual content from Shopify with custom content. When editing any field, the changes are visible but as soon as I try to save the content, switching a tab or going into the new preview mode, the changes get lost.

The shopify.product.php model is quite simple, and should not cause the problem, unless anything is missing, like e.g overwriting the save function:

<?php

use Kirby\Cms\Page;
use Kirby\Content\Field;

class ShopifyProductPage extends Page
{
  public function smartPrice(): string
  {
    $price = $this->shopifyPrice()->value();
    $compareAtPrice = $this->shopifyCompareAtPrice()->value();
    
    if (!empty($compareAtPrice) && $compareAtPrice !== '0.00') {
      return '<span class="product__price"><ins>' . $this->formatPrice($price) . '</ins> <del>' . $this->formatPrice($compareAtPrice) . '</del></span>';
    }
    
    return '<span class="product__price">' . $this->formatPrice($price) . '</span>';
  }
  
  public function smartReleaseDate(): Field
  {
    return new Field($this, 'smartReleaseDate', $this->release()->or($this->shopifyPublishedAt())->or($this->shopifyCreatedAt()));
  }
}

shopify.products.php:

<?php

use Kirby\Cms\Page;
use Kirby\Cms\Pages;
use Kirby\Data\Yaml;
use Kirby\Toolkit\Str;
use Kirby\Uuid\Uuid;

class ShopifyProductsPage extends Page
{
  static $subpages = null;

  public function subpages()
  {
    if (static::$subpages) {
      return static::$subpages;
    }

    return static::$subpages = Pages::factory($this->inventory()['children'], $this);
  }
  public function children(): Pages
  {

    if ($this->children instanceof Pages) {
      return $this->children;
    }

    $products = \KirbyShopify\App::getProducts();
    $activeproducts = array_filter($products, function ($product) {
      return $product['status'] == 'active';
    });


    $activeproductscount = count($activeproducts);

    $pages = array_map(function ($product) {
      $slug = Str::slug($product['handle']);
      $page = $this->subpages()->findBy('slug', $slug);
      $pagecontent = $page ? $page->content()->toArray() : null;

      $shopifyProduct = [
          'title'                  => $product['title'],
          'shopifyStatus'          => $product['status'],
          'shopifyTitle'           => $product['title'],
          'shopifyID'              => $product['id'],
          'shopifyCreatedAt'       => $product['created_at'],
          'shopifyUpdatedAt'       => $product['updated_at'],
          'shopifyPublishedAt'     => $product['published_at'],
          'shopifyHandle'          => $product['handle'],
          'shopifyVendor'          => $product['vendor'],
          'shopifyFeaturedImage'   => count($product['images']) > 0 ? Yaml::encode([$product['images'][0]]) : '',
          'shopifyImages'          => Yaml::encode($product['images']),
          'shopifyDescriptionHTML' => $product['body_html'],
          'shopifyPrice'           => count($product['variants']) > 0 ? $product['variants'][0]['price'] : '',
          'shopifyCompareAtPrice'  => count($product['variants']) > 0 ? $product['variants'][0]['compare_at_price'] : '',
          'shopifyType'            => $product['product_type'],
          'shopifyTags'            => $product['tags'],
          'shopifyVariants'        => Yaml::encode($product['variants']),
          'shopifyOptions'         => Yaml::encode($product['options']),
          'shopifyInventory'       => count($product['variants']) > 0 ? $product['variants'][0]['inventory_quantity'] : '',
      ];

      if ($pagecontent) {
          foreach ($shopifyProduct as $k) {
              unset($pagecontent[strtolower($k)]);
          }
      }

      if ($page && $page->num()) {
          $num = $page->num();
      } else {
          $publishedAt = $product['published_at'];
          $createdAt = $product['created_at'];
          $timestamp = $publishedAt ? strtotime($publishedAt) : strtotime($createdAt);
          $num = date('Ymd', $timestamp); // Convert timestamp to YYYYMMDD format
      }

      return [
        'slug'      => $slug,
        'num'       => $num,
        'template'  => 'shopify.product',
        'model'     => 'shopify.product',
        'files'     => $page ? $page->files()->toArray() : null,
        'uuid'      => $page ? $page->uuid() : Uuid::generate(),
        'content'   => $page ? $shopifyProduct + $pagecontent : $shopifyProduct,
      ];
    }, $products);

    usort($pages, function ($a, $b) {
        return $a['num'] <=> $b['num'];
    });

    return Pages::factory($pages, $this);
  }
}

Would be happy for any hints.

All the best
Jakob

[bastianallgeier]

Sorry for the delay. I’m currently looking into it.

[bastianallgeier]

I think I have a solid solution for you, but we will need to test this some more together and see if it really fits. Our cookbook article is unfortunately no longer valid and we need to update that too once we are sure that my solution works properly.

With our new Storage architecture, we can now create a simple mixed content storage handler. This will combine the default PlainTextStorage with some additional virtual values. The big advantage here is that we get everything out of the box for free if the storage handler is written correectly (translations, versions, etc.)

This new MixedStorageHandler is only needed once. E.g. you can create it as a plugin (site/plugins/mixed-storage/index.php) and can then be reused for multiple virtual content scenarios.

<?php
namespace Kirby\Content\MixedStorage;

use Kirby\Cms\Language;
use Kirby\Content\PlainTextStorage;
use Kirby\Content\VersionId;

class MixedStorage extends PlainTextStorage
{
    protected array $virtual = [];

    /**
     * Read the original content from disk and merge it with the virtual content
     */
	public function read(VersionId $versionId, Language $language): array
	{
        $content = parent::read($versionId, $language);

        return [
            ...$this->readVirtual($versionId, $language),
            ...$content,
        ];
	}

    /**
     * Check if the page exists on disk and otherwise check if there is any virtual content
     */
    public function exists(VersionId $versionId, Language $language): bool
    {
        return parent::exists($versionId, $language) || $this->readVirtual($versionId, $language) !== [];
    }

    /**
     * Read virtual content for a given version and language from our
     * in-memory storage array
     */
    public function readVirtual(VersionId $versionId, Language $language): array
    {
        return $this->virtual[$versionId->value()][$language->code()] ?? [];
    }

    /**
     * Write virtual content for a given version and language to our
     * in-memory storage array
     */
    public function writeVirtual(VersionId $versionId, Language $language, array $data): void
    {
        $this->virtual[$versionId->value()][$language->code()] = $data;
    }
}

With this new Storage class, it’s now actually pretty easy to create virtual pages that can still have content on disk. I’ve adjusted our example from the cookbook article, but it’s totally applicable to the shopify example above as well.

<?php

use Kirby\Content\MixedStorage;
use Kirby\Cms\Language;
use Kirby\Cms\Pages;
use Kirby\Content\Storage;
use Kirby\Content\VersionId;
use Kirby\Toolkit\A;
use Kirby\Uuid\Uuid;

class AnimalsPage extends Page
{
	public function children(): Pages
	{
		if ($this->children instanceof Pages) {
			return $this->children;
		}

		$csv   = csv($this->root() . '/animals.csv', ';');
		$pages = new Pages();

		foreach ($csv as $animal) {
			$slug = Str::slug($animal['Scientific Name']);

			// No need to check for existing pages here. We can
			// simply assume that some of them might already be on disk
			// and some of them are not there yet.
			$page = Page::factory([
				'slug'     => $slug,
				'template' => 'animal',
				'model'    => 'animal',
				'parent'   => $this,
				'num'      => 0,
			]);

			// Switch to our new storage handler to keep all the default
			// features for pages on disk, but also add the virtual content layer
			$page->changeStorage(MixedStorage::class);

			// Write virtual content to our in-memory storage array
			$page->storage()->writeVirtual(
				versionId: VersionId::latest(),
				language: Language::ensure('default'),
				data: [
					'title'       => $animal['Scientific Name'],
					'commonName'  => $animal['Common Name'],
					'description' => $animal['Description'],
					'uuid'        => Uuid::generate(),
				]
			);

			$pages->add($page);
		}

		return $this->children = $pages;
	}
}

The cool part about this solution is that we no longer need that additional subpages method and we also don’t need to check for existing pages. We can also extend this very easily to add virtual content for other versions (changes) or translations by calling $page->storage()->writeVirtual() with a different VersionId or a different Language object.

[grommasdietz]

Thanks for your solid foundation! It’s working great, except for one major issue on my multilingual site: it doesn’t work for the default language.

I’ve tried a few things already and will test it in an isolated setup next to see if I can narrow it down.

[grommasdietz]

After creating a minimal test case with Kirby Plainkit and a multilingual setup, I was able to identify the core issue. When a page with mixed content is saved, the virtual data is written to the physical text file. This is not the desired behavior, as this data should always remain virtual and be loaded fresh from its source on each request.

The definitive solution is to extend the MixedStorage handler with a custom write() method. This method intercepts the save operation and filters out the virtual fields, ensuring only the real content is ever written to disk.

    /**
     * Intercept the write process to filter out virtual data.
     */
    public function write(VersionId $versionId, Language $language, array $data): void
    {
        // 1. Get the keys of the data that has been set as virtual for the current context.
        $virtualKeys = array_keys($this->readVirtual($versionId, $language));

        // 2. Use A::without() to remove these dynamic virtual keys from the data array
        $saveData = A::without($data, $virtualKeys);

        // 3. Pass only the "real" data to be saved to the file.
        parent::write($versionId, $language, $saveData);
    }

The full MixedStorageHandler:

<?php

namespace Kirby\Content\MixedStorage;

use Kirby\Cms\Language;
use Kirby\Content\PlainTextStorage;
use Kirby\Content\VersionId;
use Kirby\Toolkit\A;

class MixedStorage extends PlainTextStorage
{
    protected array $virtual = [];

    /**
     * Read the original content from disk and merge it with the virtual content
     */
	public function read(VersionId $versionId, Language $language): array
	{
        $content = parent::read($versionId, $language);

        return [
            ...$this->readVirtual($versionId, $language),
            ...$content,
        ];
	}

    /**
     * Intercept the write process to filter out virtual data.
     */
    public function write(VersionId $versionId, Language $language, array $data): void
    {
        // 1. Get the keys of the data that has been set as virtual for the current context.
        $virtualKeys = array_keys($this->readVirtual($versionId, $language));

        // 2. Use A::without() to remove these dynamic virtual keys from the data array
        $saveData = A::without($data, $virtualKeys);

        // 3. Pass only the "real" data to be saved to the file.
        parent::write($versionId, $language, $saveData);
    }

    /**
     * Check if the page exists on disk and otherwise check if there is any virtual content
     */
    public function exists(VersionId $versionId, Language $language): bool
    {
        return parent::exists($versionId, $language) || $this->readVirtual($versionId, $language) !== [];
    }

    /**
     * Read virtual content for a given version and language from our
     * in-memory storage array
     */
    public function readVirtual(VersionId $versionId, Language $language): array
    {
        return $this->virtual[$versionId->value()][$language->code()] ?? [];
    }

    /**
     * Write virtual content for a given version and language to our
     * in-memory storage array
     */
    public function writeVirtual(VersionId $versionId, Language $language, array $data): void
    {
        $this->virtual[$versionId->value()][$language->code()] = $data;
    }
}

[tristantbg]

Nice ! Let me know when you have a better version of this plugin !

[derfabifabi]

I have the exact same problem.
I’ll stick to Kirby 4 for now

[grommasdietz]

Got it working but we have to finish some projects before we can provide a solution. I’ll inform you as soon as it’s ready.

[derfabifabi]

thank you so much!

[derfabifabi]

Just wanted to ask if there are any updates on this one.

[grommasdietz]

We still have some work to do, but we’ll try to prepare a beta version the next weeks. We’ll inform you as soon as we are done!

[derfabifabi]

Thank you!