1

We have a custom robots.txt in the root of our IIS cloud service Azure website that does not display correctly when navigating to www.oursite.com/robots.txt . A “different” robots.txt file displays containing:

User-agent: LinkChecker  
Allow: /  
User-agent: *  
Disallow: /

We’ve tried:
1. Deleting robots.txt file in root – the incorrect robots.txt file still displays.
2. Add url-rewrite rule to web.config to display custom robots.txt – correct robots.txt will display for some time, then revert to different one

Any idea what is stepping on our custom robots.txt?

Falcon Momot
  • 25,244
  • 15
  • 63
  • 92
Brian
  • 11
  • 2

1 Answers1

0

If you're using IIS management tools for maintenance, it seems that might already have it's own settings which it presumably overwrites - if you've not configured it in the appropriate place:
IIS Robots and Sitemap extension

Zael
  • 101
  • 2
  • Welcome to Server Fault! Your answer suggests a workable solution to the question is available via another website. The Stack Exchange family of Q&A websites generally frowns on this type of answer because other websites may move, get deleted, or changed. Please read [How do I write a good answer?](http://serverfault.com/help/how-to-answer) and consider revising your answer to include the steps required to resolve the issue. And don't forget to take the [site tour](http://serverfault.com/tour). – Paul Nov 16 '15 at 15:41
  • Uh, well I just provided the resource for extra information. I did answer the question of "Any idea what is stepping on our custom robots.txt?" - it just seemed to me that there might be other possible problems. – Zael Dec 03 '15 at 09:21