I’m experiencing a strange issue: When I upload a large batch of images at once in the panel, the page blueprint seems to unload, but the .txt files retain the page blueprint’s name.
I use a file blueprint to convert the image into a webp when uploading.
When I look at the folder, I can see that all the webp files have been generated, but some of the txt files of the jpg are still there. If I delete them, the page correct blueprint is loaded again in the panel.
I have already increased the memory_limit to 2048M, upload_max_filesize to 512Mpost_max_size to 1024M and this has not helped.
The images to be uploaded, in my test, are around 90MB in total, with each image being around 3.5MB (min. 2.7MB max. 4.8MB) in size, with 25 images.
I have tried different PHP versions, like 8.2 and 8.3, do difference
Here is my Dockerfile, running Kirby 4.7.2
Dockerfile
FROM alpine:latest
# Set timezone
ENV TZ=Europe/Berlin
# Install required packages (including tzdata, shadow for usermod and openssl)
RUN apk add --no-cache \
apache2 \
apache2-ssl \
apache2-utils \
ca-certificates \
php83 \
php83-apache2 \
php83-curl \
php83-dom \
php83-gd \
php83-intl \
php83-json \
php83-mbstring \
php83-simplexml \
php83-ctype \
php83-exif \
php83-fileinfo \
php83-zip \
tzdata \
composer \
nano \
shadow \
openssl && \
cp /usr/share/zoneinfo/${TZ} /etc/localtime && \
echo "${TZ}" > /etc/timezone && \
apk del tzdata
WORKDIR /var/www/localhost/htdocs
# Copy the self-signed certificate into the image
# First create the directory for certificates
RUN mkdir -p /etc/apache2/ssl
COPY certs/server.crt /etc/apache2/ssl/server.crt
COPY certs/server.key /etc/apache2/ssl/server.key
# Install Kirby CLI
RUN composer global require getkirby/cli
ENV PATH="/root/.composer/vendor/bin:${PATH}"
# Increase PHP memory limit and exec time
RUN sed -i 's/memory_limit = .*/memory_limit = 2048M/' /etc/php83/php.ini && \
sed -i 's/upload_max_filesize = .*/upload_max_filesize = 512M/' /etc/php83/php.ini && \
sed -i 's/post_max_size = .*/post_max_size = 1024M/' /etc/php83/php.ini && \
sed -i 's/max_execution_time = .*/max_execution_time = 300/' /etc/php83/php.ini
# Remove default content in the document root
RUN rm -rf ./*
# Enable necessary Apache modules:
RUN sed -i 's/#LoadModule rewrite_module/LoadModule rewrite_module/' /etc/apache2/httpd.conf && \
sed -i 's/#LoadModule headers_module/LoadModule headers_module/' /etc/apache2/httpd.conf && \
sed -i 's/#LoadModule deflate_module/LoadModule deflate_module/' /etc/apache2/httpd.conf
# Enable AllowOverride All for the document directory
COPY conf.d/allowoverride.conf /etc/apache2/conf.d/allowoverride.conf
# Copy config for deflate/gzip compression
COPY conf.d/gzip.conf /etc/apache2/conf.d/gzip.conf
# Output SSL error log to Docker output
RUN ln -sf /dev/stdout /var/log/apache2/ssl_error.log
# Copy the ENTRYPOINT script (for dynamic UID adjustment)
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
COPY app .
# Expose port 443 (HTTPS only)
EXPOSE 443
# Start Apache
CMD ["/usr/sbin/httpd", "-D", "FOREGROUND"]
When I tried ImageMagick once, I always got the error in the Panel: The new file exists and cannot be overwritten.
But with gd all images are always processed and converted to webp, what usually remains are the 1-3 .txt files from the old jpg images.
The deletion of the source happens after the conversion through Kirby, which means all images have already been successfully converted. (Or im getting this wrong?)
No matter what image set I tested it with, what was left over were always just a few .txt files from the original images, never the source images themselves.
Is there something like a limit of a number of images when this starts to happen, like works fine with 5 or 10, but starts to be problematic with 20 or so?
Are you testing this on a remote server or in your local test environment?
Edit: Just noticed you are running Alpine Linux in Docker. Maybe that’s the problem?
When I upload one or two pictures, it always works without any problems,
with 5 pictures it’s a 50/50 thing whether it works or not
With 20 pictures it basically always fails, so leaving the orphan .txt files.
Before I built the site in a container, I also had this issue, and I was using Ubuntu. But I also rebuilt the Dockerfile for testing purposes, so that it is based on debian with PHP 8.2 and there also the same problem.
Edit: The problem occurs in both of my environments, therefore on my test server and on the production server.
Kirby 5 comes with a new uploader that has the main goal of enabling chunked uploads. It may or may not improve the general reliability as well. Have you tried if Kirby 5 makes a difference to this issue in your setup?
A general idea for a possible core improvement: Instead of uploading to the content folder and doing the format conversion to WebP there, we could check during the upload whether a format conversion or resizing operation will take place. In this case we could replace the “move to content folder” step with the processing step directly, thus making the upload behavior more atomic. @distantnative
Actually, I don’t have any problems with uploading using 5.0.0-rc.5.
Since I use plugins such as Merx, the upgrade to Kirby 5 will take a little longer. This is primarily because Merx 2.0 will have several breaking changes, and it is unclear when a Merx version for Kirby 5 will be available.
@lukasbestle Is it therefore possible to make the new uploader available in Kirby 4 as well?
Okay, there are still leftover jpg.txt files in Kirby 5, but the blueprint is no longer unloads, so it is usable.
Even if a few orphan .txt files remain.
It sounds like the underlying issue still occurs with the new uploader of Kirby 5. This really shouldn’t happen, but I’m not aware of similar reports by other users. Feel free to open a GitHub issue on this. With the current set of details we will not be able to find a root cause, for that we need to be able to reproduce this issue. But maybe other people are affected by it as well and can contribute additional information to enable reproduction.
On the new uploader: We won’t be able to port that back to Kirby 4 unfortunately.
In a project I’m currently developing this bug was a real showstopper for my client who should upload project images while I was on holiday – whenever they tried to bulk upload multiple images (which got converted from jpg to webp) they got some weird panel feedback and after reloading/leaving the parent entry it was nowhere to be found again in the panel.
I just updated from rc2 to rc6, did a test myself and noticed the same behaviour – the randomly created additional jpg.txt files make the whole parent folder disappear from the panel, as soon as I manually delete the superfluous files the page reappears as expected.
These superfluous files aren’t great, but the real problem seems to me to be the disappearing pages they are currently causing. – this can be easily recreated by simply duplicating an existing image.wepb.txt file and renaming it to image.jpg.txt (expecting that an web-file with this name exists) and whoops, the whole page is gone… isn’t that something, that should be avoided/fixed independent from the bulk-conversion-issue itself?
Happens to approx. 2/5 of all bulk uploaded images and can already appear when only 2 images get uploaded. Haven’t noticed this error when uploading single images though…
I also don’t see anything in my php error logs and didn’t notice any wrong/missing converted image file itself – it’s just the superfluous txt-files, which are causing all the trouble…
I’d create a new issue on GitHub, please provide all necessary details to reproduce the issue if you can. Please also add information on installed plugins.
I think it makes sense to separate these issues (superfluous txt-files get created / superfluous txt-files make parent page dissappear) and started with a bug report for the second issue, as this one is way more urgent and annoying at the moment: