9

I just keep getting a message about

"Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%. You can see more details about these errors in Webmaster Tools. "

I searched it and told me to add robots.txt on my site

And when I test the robots.txt on Google webmaster tools ,the robots.txt just cannot be fetched. enter image description here

I thought maybe robots.txt is blocked by my site ,but when I test it says allowed by GWT.

enter image description here

'http://momentcamofficial.com/robots.txt' And here is the content of the robots.txt : User-agent: * Disallow:

So why the robots.txt cannot be fetched by Google?What did I miss .... Can anybody help me ???

Jason
  • 221
  • 2
  • 3
  • 7

4 Answers4

2

I had a situation where Google Bot wasn't fetching yet I could see a valid robots.txt in my browser.

The problem turned out that I was redirecting my whole site (including robots.txt ) to https, and Google didn't seem to like that. So I excluded robots.txt from the redirect.

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteCond %{REQUEST_FILENAME} !robots\.txt
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

More info on my blog

user57429
  • 21
  • 2
  • Interesting. On June 24th I started getting these messages as well. I didn't have a robots.txt file as I wanted the full site search... but due to these messages I created one. I have been since trying to find a resolution as no one seems to know. I did convert this site to https last year and likewise redirected the full site... so this makes sense. But I wonder why it would be an issue now? – Anthony Griggs Aug 03 '15 at 16:51
1

Before Googlebot crawls your site, it accesses your robots.txt file to determine if your site is blocking Google from crawling any pages or URLs. If your robots.txt file exists but is unreachable (in other words, if it doesn’t return a 200 or 404 HTTP status code), we’ll postpone our crawl rather than risk crawling URLs that you do not want crawled. When this happens, Googlebot will return to your site and crawl it as soon as we can successfully access your robots.txt file.

As you know having robots.txt is optional so you don't need to make one, just make sure your host would send 200 or 404 http status only.

Moein Nourmohammadi
  • 128
  • 2
  • 8
0

You have the wrong content in your robots.txt file, change it to:

User-agent: *
Allow: /

And make sure that everybody has the permissions to read the file.

spydon
  • 9,372
  • 6
  • 33
  • 63
  • 1
    Yes I too getting same messages even after allowing, Please check my [robots.txt](http://www.msccomputerscience.com/robots.txt). – ARJUN Oct 18 '14 at 09:39
  • Do you have a reason to have all the specific crawlers listed as you do in your robots.txt? Else you can try removing everything down to 'User-agent: *' and it should work. – spydon Oct 19 '14 at 18:23
0

I was getting this error when "yandex" crawled the site and also with some website checkers. After checking everything multiple times, I made a copy of robots.txt and called it robot.txt. Now "yandex" and the tool, both work.

Rohit Gupta
  • 4,022
  • 20
  • 31
  • 41