We use Oh Dear! to monitor the uptime of our client websites. In addition Oh Dear! also looks for broken links and found nearly all links on our client’s website to be broken:
Interestingly all links mentioned in Oh Dear! work perfectly in a browsers and also the Google Search Console doesn’t mention any 500 errors. The kind guys at Oh Dear! tried to access our client’s homepage via curl and then get the 500 errors:
$ curl -i "https://www.ceranews.com/to/100p5rb"
HTTP/1.1 500 Internal Server Error
Date: Sun, 03 Nov 2019 12:19:59 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
Content-Length: 530
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
webmaster@nrw.net to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
I know how to solve 500 errors in the panel but here I am totally lost. Any ideas out there?
This might be related to some security or firewall software which blocks or alters requests from unknown/uncommon user-agents or suspicious IP addresses.
I’ve debugged this a bit using Charles proxy to see what happens with requests sent from the browser.
It turns out that the HTTP 500 error occurs when no Cookie header is transmitted in the request and no browser-like user agent is passed. This only affects the tinyurl URLs though, other URLs work just fine without the cookie and user agent.
The issue is: There is no piece of code I can think of that would deliberately cause this error. Meaning: Kirby doesn’t check for the cookie and user agent anywhere as a security feature of some sorts.
So it really looks like an issue of the server configuration of your hosting provider as @warg suspected. Maybe they’ll be able to track down that error pattern. If not, they can at least take a look at the logs to find out what exactly causes it.
@yves Defintatly some weirdness going on, you might want to look into the mixed charachter encodings at the same time…(although thats probably the apache error page encoding)
i used Screaming Frog which is very useful for things like this, and will find broken links etc also. Sadly my liscence has expired and the free edition doesnt allow you to change the useragent (it can pretend to be most known crawler like the ones search engines use) and mess about with the headers but its worth the money since it does a huge amount things besides this.
