I am finding that Siteliner is unable to crawl a website I created with Kirby. Siteliner is being blocked. Can anyone suggest a solution to this?
Thanks in advance
Certainly nothing to do with Kirby. Maybe you have a robots.txt in place to block bots, or the server is somehow protected against these automated crawls.
Thank you, I’ll have another look
if you use my robots-txt plugin make sure global debug mode is disabled as explained in the readme. it will disallow crawling otherwise.
the robots.txt created by the popular SEO plugin from @tobimori will disallow indexing when the global debug option is set to true as well.