0

I have a free site that utilizes Byethost, which utilizes iFastnet hosting services. For a little while now my site has become "de-indexed" by google, because it cannot fetch any aspect of my site now for some reason. I can access my site without issues in any web browser and I have not changed any code for the site. My robots.txt is even set to

    User-agent: *
    Allow: /

Strangely enough the google structured data tool is able to fetch any page, but every other aspect of the google crawler fails to fetch resources from the site.

For example, when testing robots.txt through google, it says

    You have a robots.txt file that we are currently unable to fetch. In such cases we stop crawling your site until we get hold of a robots.txt, or fall back to the last known good robots.txt file. Learn more.

So clearly google is getting blocked somehow before it can even reach the server and download the robots.txt resource.

When I cURL the site from my computer, it also hits me with a 403 Error. But when I try to cURL a copy of the site on a private server, it returns the page without issues. Surely this must be related to Google's issue.

But I have no .htaccess or any security features like IP blacklisting enabled, so this is super confusing.

My free plan with byethost should also not have cloudflare as it is a premium feature.

Can anyone provide some insight as to why I am getting these 403 errors? Is this the same reason why Google cannot access the site anymore? How can I fix this?

Thank you

gunter
  • 139
  • 5
  • What happens when you visit your site's robots.txt in the browser? – ceejayoz Sep 27 '19 at 14:42
  • Displays the robots.txt file without issues. – gunter Sep 27 '19 at 14:45
  • 1
    If you can't get at it via cURL, but you can in the browser, that seems to clearly indicate that there's some sort of user-agent block in play. Contact your host. – ceejayoz Sep 27 '19 at 14:47
  • I have free hosting so I am not sure how much they can do for me :(. Surely they wouldn't block google requests though? And as mentioned the google structured data tool is strangely able to fetch pages. – gunter Sep 27 '19 at 14:49
  • 1
    Anything's possible with free hosting. – ceejayoz Sep 27 '19 at 15:06

0 Answers0