Bulk Builder field to Kirby Block 3.7 conversion

Hey there,

the transition from Builder field to Kirby Blocks was smooth until Kirby v3.6. In the latest version (3.7) the automatic migration is gone, which is understandable.

Unfortunately I have a Kirby website with a lot of pages that all use the Builder field and that need migration to Kirby Blocks. Changing the blueprints and templates/snippets is not a big deal, but having the data saved properly so that also v3.7 can read it, is.

If I understand correctly, until v3.6 the data that was saved from the Builder field was converted to the Kirby-Block-(JSON-)Syntax on saving of pages. At the moment for me this means, that I would have to go through hundreds of pages manually, change something, save it, in order to have all my data stored in the new Kirby-Block-Syntax so that v3.7 can actually work with it.

Is there a way to convert the data (the field value that are stored in the txt-files) programmatically / in one automated process? Or is there a thing or an approach that I missed?

I hope I made the issue understandable. Thanks in advance. Help is very appreciated. :slight_smile:

Yes, this should be possible programmatically, using the old BlockConverter class from until 3.6 as a basis. Quoting what @distantnative outlined on Discord:

Thinking out loud how to approach this:

  • take BlockConverter class and put it in a plugin
  • in the plugin loop through all blueprints and find all fields with type for old blocks
  • for each blueprint/template loop through whole content index and run fields that have been identified as potential old blocks through converter
  • update content files with results from converter

@texnixe Thank you very much for the reply (and thanks @distantnative for the suggestions).

Yes, I also was thinking that the BlockConverter class should be capable of this, but having a look at the builderBlock method made me hesitant since I couldn’t see any YAML to JSON conversion here - but I probably don’t understand the architecture here completely.

So I am going to try this, but I am still a bit lost in terms of how to use the BlockConverter on a field / page. So I guess I should be using BlockConverter::builderBlock() but I have no real idea from where to get the $params before or how to create/save the new converted field based on the $params I get returned.

Any more hints of where I should look into maybe?

I’ve never used the builder, only the editor, so not 100% positive my script also works for the builder migration, but you can give it a try! Add the following to your config.php, then (after you changed the field type to blocks in your blueprints) visit https://yoursite.com/blocks-migration :slight_smile:

use Kirby\Cms\BlockConverter;
use Kirby\Cms\Response;
use Kirby\Toolkit\A;

return [
  'routes' => [
    'pattern' => '/blocks-migration',
    'action' => function() {
      if (!kirby()->user() || !kirby()->user()->isAdmin()) {
        return new Response('You need to log in as admin.', 'text/plain', 401);
      }

      $csrf = csrf();
      return "
        <!doctypehtml><meta charset=UTF-8><h1>Editor → Blocks</h1>
        <form method=POST>
          <input type=hidden name=csrf value=$csrf>
          <button>Start migration</button>
        </form>
      ";
    }
  ],
  [
    'pattern' => '/blocks-migration',
    'method' => 'POST',
    'action' => function() {
      $kirby = kirby();

      if (!$kirby->user() || !$kirby->user()->isAdmin()) {
        return new Response('You need to log in as admin.', 'text/plain', 401);
      }
      if (!csrf(get('csrf'))) {
        return new Response('Invalid token.', 'text/plain', 403);
      }

      $pages = site()->index(true);
      $updated_contents = [];

      foreach($pages as $page) {
        $block_fields = A::filter($page->blueprint()->fields(), fn($f) => $f['type'] === 'blocks');

        foreach($block_fields as $field) {
          $name = $field['name'];

          if (!$kirby->multilang()) {
            if (!$page->content()->has($name)) continue;

            $value = $page->content()->get($name)->toData('json');
            $converted = BlockConverter::builderBlock($value);
            if (!A::isAssociative($converted)) {
              $converted = BlockConverter::editorBlocks($converted);
            }

            if ($value === $converted) continue;

            $page->update([ $name => $converted ]);
            $updated_contents[] = "{$page->id()} → $name";
          } else {
            foreach($kirby->languages() as $lang) {
              $langCode = $lang->code();

              if (!$page->content($langCode)->has($name)) continue;

              $value = $page->content($langCode)->get($name)->toData('json');
              $converted = BlockConverter::builderBlock($value);
              if (!A::isAssociative($converted)) {
                $converted = BlockConverter::editorBlocks($converted);
              }

              if ($value === $converted) continue;

              $page->update([ $name => $converted ], $langCode);
              $updated_contents[] = "{$page->id()} → $name ($langCode)";
            }
          }
        }
      }

      $count = count($updated_contents);
      return "
        <!doctypehtml><meta charset=UTF-8><h1>Editor → Blocks</h1>
        <p>Migrated $count field contents to the native blocks format.
        <ol>". r($count, '<li>') . A::join($updated_contents, '<li>') . "</ol>
        <a href={$kirby->site()->panelUrl()}>Back to the panel</a>
      ";
    }
  ]
];

If you have a loooooot of pages and fields, you might need to update the script to use batched processing as described here in the cookbook: Batch updating content | Kirby CMS

4 Likes

Wow, thx a lot. I will give this a try and let you know!

Please do! I intend to polish it a little and then publish it as a Kirby plugin, so some feedback about whether it works for builder users would be super valuable :slight_smile:

Oh, wow - really nice!

I’m trying to migrate my Kirby 3.6 site to Kirby 3.7, and also make the move from builder to blocks.

When I add the code by @jonaskuske to my config.php, I get a “Cannot use object of type Closure as array” error, though – do you have any ideas on what causes this?

You’ve probably saved the Builder values as yaml so you need to replace json with yaml in the toData() methods.

We’ve adopted the above script to be executed as a CLI script and also reimplemented the convert function so it also works with Kirby 3.7. But this version lacks Support for converting Editor fields.

To run this, place this as convert_builder.php and run it with php convert_builder.php in the root of your Kirby installation. Please note: To convert nested Builder blocks you need to run this script with Kirby 3.6.

<?php

use Kirby\Toolkit\A;

require 'kirby/bootstrap.php';
$kirby = new Kirby();

$pages = site()->index(true);
$updated_contents = [];

function convertBuilderBlock(array $params): array
{
    if (isset($params['_key']) === false) {
        return $params;
    }

    $contentOut = [
        'type' => $params['_key'],
    ];

    unset($params['_uid']);
    unset($params['_key']);

    $contentOut['content'] = $params;

    return $contentOut;
}

function migrateField($page, $name, $langCode = null)
{
    if (!$page->content($langCode)->has($name)) {
        return false;
    };

    $blocks = $page->content($langCode)->get($name)->toData('yaml');
    $converted = [];

    foreach ($blocks as $block) {
        $converted[] = convertBuilderBlock($block);
    }

    if ($blocks === $converted) {
        return false;
    };

    $page->update([ $name => $converted ], $langCode);
    echo "Updated {$page->id()} → $name ($langCode)\n";
    return true;
}

$kirby->impersonate('kirby');
$count = 0;

foreach ($pages as $page) {
    $block_fields = A::filter($page->blueprint()->fields(), fn($f) => $f['type'] === 'blocks');

    foreach ($block_fields as $field) {
        $name = $field['name'];

        if (!$kirby->multilang()) {
            migrateField($page, $name);
            $count++;
            continue;
        }

        foreach ($kirby->languages() as $lang) {
            $langCode = $lang->code();

            migrateField($page, $name, $langCode);
            $count++;
        }
    }
}

echo "Migrated $count field contents to the native blocks format.\n";

Oh, that’s interesting!

I tried running you CLI PHP script on my 3.6.1.1 version of Kirby, but it throws this error:

Error: Call to undefined method Kirby\Toolkit\A::filter() in file /Users/fh/htdocs/fh/convert_builder.php on line 55

Do you have any ideas what causes this?

You need to upgrade to at least Kirby 3.6.5, the A::filter method was not available before: A::filter() | Kirby CMS

Hi, thanks @mactux
I tried your script on 3.6.6.1 and 3.7.5 and i have this error :

TypeError: convertBuilderBlock(): Argument #1 ($params) must be of type array, string given, called in path/convert_builder.php on line 56 in file /path/public/convert_builder.php on line 28
Stack trace:
  1. TypeError->() /path/convert_builder.php:28
  2. convertBuilderBlock() /path/convert_builder.php:56
  3. migrateField() /path/convert_builder.php:78

Do you have any idea ?

You should maybe print the value of $block. I think some conversion from yaml to array did not work. Maybe some old data inside this block or the blocks are saved as json? Then you need to change

$blocks = $page->content($langCode)->get($name)->toData('yaml');

to

$blocks = $page->content($langCode)->get($name)->toData('json');

I’ve changed yaml to json and the command run correctly, but didn’t fix the blocks :unamused:

I have 2 types of structure

the old one that not working
Was created with the editor plugin

{
        "attrs": [],
        "content": "The content here",
        "id": "_6a9p0habb",
        "type": "paragraph"
    },

The recent that works

{
        "content": {
            "text": "the content here"
        },
        "id": "1e853828-b5dc-4e2b-a358-7e9835f6b65d",
        "isHidden": false,
        "type": "text"
    },

So, i think, i have to looking for migration editor to blocks…

@mactux Thanks again

Jonas thank you for sharing your editor-migration script.

I modified it to add support for block fields inside of structure fields:

use Kirby\Cms\BlockConverter;
use Kirby\Cms\Response;
use Kirby\Toolkit\A;

return [
	'routes' => [
		[
			'pattern' => '/blocks-migration',
			'action' => function() {
				if (!kirby()->user() || !kirby()->user()->isAdmin()) {
					return new Response('You need to log in as admin.', 'text/plain', 401);
				}

				$csrf = csrf();
				return "
					<!doctypehtml><meta charset=UTF-8><h1>Editor → Blocks</h1>
					<form method=POST>
						<input type=hidden name=csrf value=$csrf>
						<button>Start migration</button>
					</form>
				";
			}
		],
		[
			'pattern' => '/blocks-migration',
			'method' => 'POST',
			'action' => function() {
				$kirby = kirby();

				if (!$kirby->user() || !$kirby->user()->isAdmin()) {
					return new Response('You need to log in as admin.', 'text/plain', 401);
				}
				if (!csrf(get('csrf'))) {
					return new Response('Invalid token.', 'text/plain', 403);
				}

				$pages = site()->index(true);
				$updated_contents = [];

				foreach($pages as $page) {
					$block_fields = A::filter($page->blueprint()->fields(), fn($f) => $f['type'] === 'blocks');

					foreach($block_fields as $field) {
						$name = $field['name'];

						if (!$kirby->multilang()) {
							if (!$page->content()->has($name)) continue;

							$value = $page->content()->get($name)->toData('json');
							$converted = BlockConverter::builderBlock($value);
							if (!A::isAssociative($converted)) {
								$converted = BlockConverter::editorBlocks($converted);
							}

							if ($value === $converted) continue;

							$page->update([ $name => $converted ]);
							$updated_contents[] = "{$page->id()} → $name";
						} else {
							foreach($kirby->languages() as $lang) {
								$langCode = $lang->code();

								if (!$page->content($langCode)->has($name)) continue;

								$value = $page->content($langCode)->get($name)->toData('json');
								$converted = BlockConverter::builderBlock($value);
								if (!A::isAssociative($converted)) {
									$converted = BlockConverter::editorBlocks($converted);
								}

								if ($value === $converted) continue;

								$page->update([ $name => $converted ], $langCode);
								$updated_contents[] = "{$page->id()} → $name ($langCode)";
							}
						}
					}

					$structures = A::filter($page->blueprint()->fields(), fn($f) => $f['type'] === 'structure');

					foreach($structures as $structure) {
						$structureName = $structure['name'];

						if (!$kirby->multilang()) {
							if (!$page->content()->has($structureName)) continue;

							$structureContent = $page->content()->get($structureName)->yaml();

							foreach($structureContent as $i => $structureRow){
								$block_fields = A::filter($structure['fields'], fn($f) => $f['type'] === 'blocks');

								foreach($block_fields as $field){
									$name = $field['name'];
									if (!array_key_exists($name, $structureRow)) continue;

									$value = Data::decode($structureRow[$name], 'json');
									$converted = BlockConverter::builderBlock($value);
									if (!A::isAssociative($converted)) {
										$converted = BlockConverter::editorBlocks($converted);
									}

									if ($value === $converted){
										echo "nothing changed in {$page->id()} → $structureName → $name<br>";
										continue;
									}

									$structureContent[$i][$name] = Data::encode($converted, 'yaml');
									$updated_contents[] = "{$page->id()} → $structureName → $name";
								}
							}

							$page->update([ $structureName => $structureContent ]);
						} else {
							foreach($kirby->languages() as $lang) {
								$langCode = $lang->code();

								if (!$page->content($langCode)->has($structureName)) continue;

								$structureContent = $page->content($langCode)->get($structureName)->toStructure();

								foreach($structureContent as $i => $structureRow){
									$block_fields = A::filter($structure['fields'], fn($f) => $f['type'] === 'blocks');

									foreach($block_fields as $field){
										$name = $field['name'];
										if (!array_key_exists($name, $structureRow)) continue;

										$value = Data::decode($structureRow[$name], 'json');
										$converted = BlockConverter::builderBlock($value);
										if (!A::isAssociative($converted)) {
											$converted = BlockConverter::editorBlocks($converted);
										}

										if ($value === $converted){
											echo "nothing changed in {$page->id()} → $structureName → $name<br>";
											continue;
										}

										$structureContent[$i][$name] = Data::encode($converted, 'yaml');
										$updated_contents[] = "{$page->id()} → $structureName → $name ($langCode)";
									}
								}

								$page->update([ $structureName => $structureContent ], $langCode);
							}
						}
					}
				}
				

				$count = count($updated_contents);
				return "
					<!doctypehtml><meta charset=UTF-8><h1>Editor → Blocks</h1>
					<p>Migrated $count field contents to the native blocks format.
					<ol>". r($count, '<li>') . A::join($updated_contents, '<li>') . "</ol>
					<a href={$kirby->site()->panelUrl()}>Back to the panel</a>
				";
			}
		]
	]
];

I tested it in a single-language environment.

Thank you very much for posting this. I have used it succesfully to migrate editor (now obsolete plugin) field into blocks.

I had to downgrade kirby core to 3.6 to be able to make it work, which felt a bit awkward.