How do I know if redis is working well? Or, what do to against AI scrappers sucking hosting power?

Hi there,

That may sound as a simple question.. At the moment I simply added Redis as driver for my pages cache but how do I know if it’s actually well working on my prod server? It’s a managed one (infomaniak) and all I know is that Redis is installed.

It’s a bit of a black box right now and I’d love to know if all is working fine.

Also, in Kirby’s reference page about cache it says that I can also cache on uuid but not sure how important is that and if I should do it or not.

Thanks for your help!

Just my 2c… unless you have more than 4k pages using Redis will not measurably improve your performance since the server will keep the content files and UUID cache files “hot” in disk cache RAM and thus as quick as Redis.

In regard to Redis you can create custom plugin, make it have a cache living on Redis and then create and read cache values.
Or if your host provides you with an URL use a tool like TablePlus.

Thank for your quick answer. We’re more around 1k pages for the moment.. Do you suggest another cache driver?

memcache and APCu are supposedly available on infomaniak also. We’re with a 4vCPU, 12Go RAM and 250 Go ssd nvme (first offer).

This is already a step toward more power as we were using the mutual offer at 5€/month.

Still, my uptime monitor catch regularly 503 errors. And if I check the logs error, I get many :

The timeout specified has expired: AH00941: FCGI: failed to acquire connection for (babelica.alliance-publishers.org:8000)

Monitoring ressources doesn’t look much good neither, isn’it?

And here’s what I find when looking in log access:

Looks like bots are querying crazy specific searches into our books collection, maybe that’s the source of all this?

Is caching even the good angle to work on this?

I have no experience with Redis but regarding the bot traffic: the last 1k+ pages site I launched also saw tremendous traffic spikes caused by (AI) scraper bots. So I added an Nginx server directive that checks the user agent string against a list of known bots and drops the connection (444) on a match. Still evaluating but so far it seems to be working.

Thank you @phm Infomaniak managed cloud is using Apache, I’m gonna check if I have access to such settings.

Meanwhile, I’m trying with this kriby plugin GitHub - mauricerenck/dark-visitors: Dark Visitors blocks unwanted AI Crawlers from your website using robots.txt. It uses the Dark Visitors API to identify and block unwanted visitors. · GitHub by Maurice Renk.

That could help, but I find a lot of the bad faith crawlers ignore robots.txt. I’m also seeing crawlers use stealth methods like using an MS Edge user agent string, have a delay of a few seconds between each request and do batch runs of 100 requests, against which even my method doesn’t work.

Yes I’m not surprised unfortunately. Thanks for caring. I’ll wait for a few days and see accordingly what’s next move.