0

Our robots.txt file contains only this:

User-Agent: *
Disallow: /

However, we've been receiving the below errors frequently. Not sure why both Bingbot and Googlebot are looking for a Robots.txt folder 

IP: 207.46.13.42
HTTP status code: 404
URL: http://www.website.net/robots.txt/
User Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

The following error occurred:

The controller for path '/robots.txt/' was not found or does not implement IController.

IP: 66.249.74.48
HTTP status code: 404
URL: http://www.website.net/robots.txt/
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

The following error occurred:

The controller for path '/robots.txt/' was not found or does not implement IController.

We can code around this, but the question is why these major bots are trying to search for this folder? The goal of our robots.txt file is to prevent bots from crawling the site. From every example that exists on the internet we have set up the robots.txt file correctly to accomplish this but we still run into these bots trying to access a folder that does not exist and MVC tries to handle it causing an exception.

Michael Merrell
  • 1,006
  • 8
  • 21
  • 1
    If you request the file yourself, does it work correctly? I'm wondering if perhaps it's your routes that are not working properly and requests to the file are being redirected by .NET? – James Thorpe Nov 13 '15 at 16:00
  • They're not. They'd be looking for `website.net/robots.txt` You've probably got a bad rewrite sending them to this non-existent folder – Marc B Nov 13 '15 at 16:01
  • I am able to access the robots.txt file just fine at the root of our domain. So it is not being routed to that folder at least not by any means that I can tell why it would be. Here is the robots.txt file on accessed via Chrome: https://goo.gl/photos/VeQdzeSg8djxmEfk7 – Michael Merrell Nov 13 '15 at 17:16

0 Answers0