Image generation - Image Gallery - Image Upload

1

I’m creating a frontend where image galleries are a key component.

Several issues have arisen in recent days.

I use UBERSPACE for my webspace.

I tried to upload a large number of images through the panel.

Approx. 200 images
File size up to 20 MB per image

I’ve correctly configured the server to allow uploads of such large files.

Nevertheless, the upload repeatedly breaks off after about 20 images and displays an error in a popup window:

Filename.jpg ERROR uploading image <<<

→
Questions:
Is the Kirby Panel not designed for such large images in such large quantities?
Could this be a bug in the panel?
Upload via SFTP works perfectly of course!

Since I couldn’t upload the images in their full size, I reduced the size of the images on my local computer. The file size is now up to 1MB! The upload now works without a hitch.

BUT

In my frontend, the images are displayed in a list. There are 2 images side by side. As soon as an image comes into the user’s field of vision, it loads. So, if you quickly scroll through the list, my frontend makes about 200 requests to the server for icons.

This brings the server to a halt.

The panel displays the following error in a popup:

The JSON response could not be parsed
Gateway Timeout
The gateway did not receive a timely response from the upstream server or application. <<<

Bildschirmfoto 2023-08-28 um 05.27.00
Bildschirmfoto 2023-08-28 um 05.27.001302×400 32.1 KB

When the server is in this state, which lasts for several minutes, it no longer responds. Neither the panel opens, nor can my app make requests. Such behavior is of course unacceptable for a commercial application.

Of course, I could now limit the server request, so request one icon after another. This would work as long as there is only one app or website user. However, if multiple users access a new gallery simultaneously, it forces the server temporarily to its knees.

One solution would be for me to enter somewhere in the panel which image sizes I need and then have them automatically generated before the gallery is published. BUT this would have to be a global setting. My app requires about 200 images 3 times a week. Manually entering the size for each image 600 times would be too time-consuming for my clients.

→
Questions:
Is the server’s hangup due to my webspace, is Uberspace not suitable for an application of this type?
If so, which webspace can you recommend?

Bildschirmfoto 2023-08-28 um 06.49.17
Bildschirmfoto 2023-08-28 um 06.49.171606×3260 791 KB

My images are located in the panel in the files section. 20 images are displayed per page. I use the Cards / Tiny view. Dividing them into groups of 20 images makes sorting an image gallery of 200 images quite complicated.

→
Question:
Is there a setting I haven’t found that displays all the images so you can easily drag and drop the images?

Bildschirmfoto 2023-08-28 um 06.25.50
Bildschirmfoto 2023-08-28 um 06.25.501718×2212 220 KB

2

What exactly did you configure? I think it’s very likely that you run into script runtime errors with so many and such huge uploads. And the larger your original image, the more computing power is needed when resizing such images.

200 images with 20MB seems a bit much, no matter what (i.e. waste of resources, because you very likely don’t want to display such huge images).

3

upload_max_filesize = 128M

post_max_size = 132M

memory_limit = 256M

max_execution_time = 9000

max_input_time = 9000

This are my settings.

But as I wrote I just uploaded much smaller pictures so the upload worked. Anyway the Server haltet for a while to compute the smaller images from the 1MB Pics.

Is uberpace not the best option for a project like mine?

4

That really depends very much on your needs. If your server/provider does not provide all the resources you need I propose you start optimising with thumbnails, webP, caching and other steps to keep traffic on the low side. Optimizing images beforehand is an even better idea - if you have the capacities and knowledge and want to carefully control the output. But big data will eventually clog up your server instance, either by using up all your memory or by limiting you and your visitors to work with it. Remember: Not only the server but also the visitor’s device must be able to handle such amounts of data.

5

Thank you for your answer!

Maybe my post wasn’t clear enough!

The 20MB pictures where just a test.

At the end I use pictures around 1MB in size and even with this pictures when the server has to generate 200 thumbnails it hangs.

My customers are Clubs. They upload around 100 - 300 pictures per opening day.

At the next day there is a lot of traffic on the pages because people search for their pictures.

6

Wow. If we really talk about 200 thumbs per 1 image I strongly suggest you have them generated using a queue. here you store the uploaded images and have then a cronjob processing the images one by one before any visitor can see them. and tbh: 200 thumbs for 1 image do sound a bit steep, maybe you can size down here?

7

Uberspace is shared hosting, with all the limitations that comes with it (although you have pretty much freedom there compared to other shared hosting).

However, have you checked with them that the PHP ini settings you have chosen are even supported on those servers? I somehow doubt it. Also, you might run into disk space issues sooner or later.

In any case, it is vital that you pregenerate those thumbs (ideally via a CLI script) with your default settings, and use lazy loading. I’d also consider pagination.

8

And one more thought / my 2 cents: How do you make sure, your customers will follow your 1MB rule? Most users have no idea about their data size and simply push it to whereever they can. If these clubs must downsize these image beforehand your service becomes less attractive as they have to put in additional work themselves.

1 Like
9

I’m the owner of the 8 clubs why I’m building the app in the first place.

The problem is at the moment it does not work with 1MB pictures so if this works I will go the next step.

I know now my first post was way too long!

The biggest problem in the moment is the thumbnail generation without killing the server when the source pictures are 1MB in size

10

Do the uploaded images also have to be in the maximum resolution? Should the images also be downloaded by the visitors? Is a reduced resolution sufficient for this?
Have you looked at plugin “Autoresize”? Maybe this is a step to the desired solution.

1 Like
11

The problem is the following:

In my frontend, the images are displayed in a list. There are 2 images side by side. As soon as an image comes into the user’s field of vision, it loads. So, if you quickly scroll through the list, my frontend makes about 200 requests to the server for icons.

This brings the server to a halt.

So what I would need is an automated way to generate all needed resolutions of my pictures:

240 x 240 | 480 x 480 | 960 x 960 | 1920 x 1920

and save them to the media (cache) folder!

OR(!)

Someone can tell me if just my server (webspace) ist to slow and I’ve to invest in a better one. So everything works as I’ve expected it … just on the fly.

Thank you for the link, uploading the images in the wanted resolution ist not a problem for me.

12

I’d create a script that pre-generates the thumbs, ideally you would do this via a CLI script to prevent running into PHP limits.

1 Like
13

This sounds great. Can you share a link to the script. Do you think a very fast server could solve my problem?

14

No, I don’t have such a script, but basically, loop through the files just as you would in your template and create the thumbs. Bootstrapping Kirby in a script, see example How to migrate users | Kirby CMS

15

Just a question what is the difference when a script generates the thumbs or my app. I could run it slower?

16

This plugin also offers a job to pregenerate thumbs: GitHub - bnomei/kirby3-janitor: Kirby 3 Plugin for running commands like cleaning the cache from within the Panel, PHP code, CLI or a cronjob

A CLI script doesn’t have the same limitations as running stuff in the browser.

1 Like
17

Ok, i understand!

Have you ever uploaded 200 images in a gallery using Kirby? Did you run in the same issue when the thumbs are generated? The source images are not over 1MB in size. If you had the same problems at least I know changing the provider does not help.

18

No

1 Like
19

ok … thank you.

20

I would suggest looking into a different/beefier hosting setup or outsource image generation to another service in general. But I can’t really tell you what to go for. There are too many variables. In the end it would come down to testing different server sizes/configurations with your intended setup.

Personally I would outsource the image generation. I used a self hosted https://imgproxy.net instance on a VPS for a image heavy site before which worked great. There are also a lot paid services to pick from. The Kirby site for example uses KeyCDN for that and you could also snag their implementation to make it work without too much hussle.

Edit: I knew I had some code snippet for generating thumbnails somewhere. Perhaps this can help as a starting point. Kirby Thumbnails · GitHub

1 Like