Is there a way to disallow robots crawling through IIS Management Console for entire site

Question

Can I do the same as robots.txt through IIS settings? Telling

User-agent: * Disallow: /

in host header or through web.config?

"Because of deployment issues" - using an approach which would prevent access (robots.txt is just a polite request) will not be possible by your criteria. The obvious way to prevent access would be to resitrict to a whitelist of IP addresses. But that's at least as difficult to deploy as a robots.txt file — symcbean, Sep 19 '11 at 14:26

score 2 · Accepted Answer · answered Sep 19 '11 at 13:08

2

No this is not a specified way. Either have the robots.txt or the HTML-Meta-Tag as described in http://en.wikipedia.org/wiki/Meta_element#The_robots_attribute

There are only these two options available.

answered Sep 19 '11 at 13:08

mailq

17,023
2
37
69

score 0 · Answer 2 · answered Sep 19 '11 at 15:32

You could use a single robots.txt file stored somewhere on your web server, and redirect all requests to any site hosted on your IIS server for robots.txt to that file. That would be a change per site in IIS, but you may be able to do it programmatically with PowerShell.

Is there a way to disallow robots crawling through IIS Management Console for entire site

2 Answers2