Google is indexing most of the pages in my website. But I’ve realised that Bing, Yahoo and DuckDuckGo are no longer indexing my site. I’m confident they were before I moved to Kirby, but they aren’t now.
I’ve signed up to Bing Webmaster Tools and done a “Live URL” Inspection of a few pages. I get the following error messages:
URL cannot be indexed by Bing
The Inspected URL returned HTTP 403 error when we tried to fetch the content. Please make sure that this is intentional. If you have moved the page to a new location, please use proper redirects.
With the following HTTP Response
HTTP Response - 403
Cache-Control : private, no-cache, no-store, must-revalidate, max-age=0
Date : Sun, 04 Jun 2023 15:14:21 GMT
Pragma : no-cache
Content-Length : 699
Content-Type : text/html
alt-svc : h3=“:443”; ma=2592000, h3-29=“:443”; ma=2592000, h3-Q050=“:443”; ma=2592000, h3-Q046=“:443”; ma=2592000, h3-Q043=“:443”; ma=2592000, quic=“:443”; ma=2592000; v=“43,46”
My .htaccess file does have redirects. Redirecting http to https and non www to www. I’ve updated my .htaccess code half an hour ago to the following. But Bing still can’t crawl the page.
# Kirby .htaccess
# revision 2022-09-15
# rewrite rules
<IfModule mod_rewrite.c>
# enable awesome urls. i.e.:
# http://yourdomain.com/about-us/team
RewriteEngine on
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule (.*) https://www.%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
# make sure to set the RewriteBase correctly
# if you are running the site in a subfolder;
# otherwise links or the entire site will break.
#
# If your homepage is http://yourdomain.com/mysite,
# set the RewriteBase to:
#
# RewriteBase /mysite
# In some environments it's necessary to
# set the RewriteBase to:
#
# RewriteBase /
# block files and folders beginning with a dot, such as .git
# except for the .well-known folder, which is used for Let's Encrypt and security.txt
RewriteRule (^|/)\.(?!well-known\/) index.php [L]
# block all files in the content folder from being accessed directly
RewriteRule ^content/(.*) index.php [L]
# block all files in the site folder from being accessed directly
RewriteRule ^site/(.*) index.php [L]
# block direct access to Kirby and the Panel sources
RewriteRule ^kirby/(.*) index.php [L]
# make site links work
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*) index.php [L]
</IfModule>
# pass the Authorization header to PHP
SetEnvIf Authorization "(.+)" HTTP_AUTHORIZATION=$1
# compress text file responses
<IfModule mod_deflate.c>
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE text/javascript
AddOutputFilterByType DEFLATE application/json
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript
</IfModule>
Could this be a problem with my redirects?