Using Google Cloud Storage for the content folder

Hello!

We’d like to host Kirby on Google AppEngine. As there can be multiple instances (if allowed via config) but also restarts to an instance, the complete content folder needs to live in a place that isn’t affected from a restart / respawn.

Google suggests Google Cloud Storage for that. It isn’t possible to use standard file commands as Kirby uses. They require to use a Google tool to access the files. More information on how to use here: https://cloud.google.com/appengine/docs/standard/php7/using-cloud-storage

What would be the best approach for this problem? Build a plugin that handles the access?

Regards,
Robert.

1 Like

as far as i know:

simply moving the content to a cloud storage is not possible in kirby. kirby 3 does not have abstraction layer like the phpleague flysystem.
to support it easily this would require big changes to kirbys core.

also the content folder itself is not enough, public media folder for thumbs, cache and site folder (license) should be considered as well.

but you can create a page model and use virtual pages to read and write to any storage you like but you also need to provide implementations for the children() method. pretty much like the virtual pages from database cookbook.

Thanks for the feedback! Okay, I suppose it’s easier to just use a database then. Didn’t think about the thumbs and cache folders yet, good point!

I’ll think a bit more about this. Kirby is so nice but the company I work at uses GCloud a lot. I hope they can both somehow fit together. :slight_smile:

I think it should be possible to mount a Google Cloud Storage Bucket into the file system so it behaves just like a normal folder to Kirby via Cloud Storage FUSE: https://cloud.google.com/storage/docs/gcs-fuse

Sounds like a good solution to host Kirby in a “Compute Engine” surrounding, where you control the whole system. But App Engine is a product on a higher level of abstraction and doesn’t give you control over the system aside from the runtime environment for PHP, etc. So I think using Cloud Storage FUSE in App Engine is not possible.

Thanks for the Feedback! So it doesn’t seem like a match of tech. The thinking behind both are too different. It came to my mind that the accounts folder should be moved to Cloud Storage, too.

We’ll elaborate a bit more on what to use.

Here are the results of my research into this topic:

When moving a CMS into the Google Cloud Platform, there are three main products to be considered for such a project, ordered by the amount of abstraction, starting with the lowest:

Google Compute Engine

  • It’s a regular VM in the cloud
  • Need to update, configure OS, runtime, etc
  • Has persistent storage built in
  • Google Cloud Storage can be connected via Filesystem (GCSFuse)

Running either a database backed or flat file CMS on Compute Engine is very easy. You start a new machine, set up your runtime environment (e.g. PHP) and your network ports and you’re good to go. You can set up a DBMS on the same machine, use another Compute Engine instance or use a cloud database (like Cloud SQL). This will work with all existing CMS, but you’re not gaining anything in terms of abstraction: It’s just like running your own server – you still need to update the operating system and runtime environment and take care of security.

Google Cloud Run

  • Runs docker containers in the cloud
  • Has no filesystem persistent storage options

Google Cloud Run is a product to run single containers in the cloud. This gives you the advantage that you can select pre-configured docker images that come with a secure and correct runtime configuration for your CMS and you can just take care of installing your CMS. However this approach is problematic: CMS typically need to be able to write to the filesystem – even database backed systems use the filesystem to store configuration, cache and session data. Google Cloud Run does not offer any persistent storage options.

Google App Engine

  • Runs application in the cloud (no container)
  • No need to configure OS or runtime
  • Has no filesystem persistent storage options
  • BUT has PHP Stream Wrappers

Google App Engine seems like the most promising candidate to run a CMS in the cloud, because its trying to abstract on the “App” level. So for App Engine we don’t need to care about the OS – we just throw our App code at it and it’s supposed to work out of the box – unless you need persistent storage. Just like Cloud Run, there are no persistent storage options in App Engine. Not only is storage not persistent, but you can’t even write to the local filesystem.

But App Engine does support PHP as a runtime environment, and PHP has a great feature for situations like this: Stream Wrappers. They allow us to override PHP file system methods like fopen() or fread() to work on other storage systems, not only the local file system. And luckily, Google offers a stream wrapper implementation for reading and writing to Google Cloud Storage buckets.

So theoretically, by using the mentioned stream wrapper, it should be possible to run a flat file CMS (or even a database backed CMS – when using an external database server) on App Engine. However it turns out, that it’s not that easy – at least not for Kirby.

Using stream wrappers, MOST filesystem methods can be adjusted to work on GCS buckets, but not all of them. One of the unsupported methods is PHPs realpath() method, which returns a canonicalized path (removing relative parts like ‘.’ and ‘…’). It is used heavily by Kirby CMS (31 times), and it will always return false when used with the GCS stream wrapper – which brings the CMS to a crash.

To make this work, one would have to override the realpath() function to return sane values. This would require changes to the Kirby codebase itself. Unfortunately Kirby uses both, its own realpath() wrapper in the F class inside the Toolkit, but also the native function in a couple of other places. So changing this behavior would require changes to the Kirby codebase in multiple places, which will make updates more complicated, etc. So it’s probably not a good idea for most people.

Also, if we could get the stream wrapper approach to work, there might be more non-compatible file system methos. This was just the first one that I came upon, which did not work. Even if it worked, it would still be problematic, because the GCS access is pretty slow. So we would definitely also need a non-filesystem caching solution (Redis seems like the best idea, because it’s available in the Google cloud).

3 Likes

I worked a little more on this topic and decided to create a little “blog post” with some code to create a prototype for this topic: GitHub - ubilabs/kirby-object-storage-stream-wrappers: Prototype to run a Kirby CMS instance with data on Object Storage (GCS)

3 Likes

Today I have seen this:

Do they solve some problems?

Are you guys still experimenting on that? I’d like to join discussions and see what we can achieve. It’s too bad, that currently there seems to be no solution. Running a docker container and mounting remote nfs could be another option we might try.