0

A bit confused with robots.txt.

Say I wanted to block robots on a site on a Linux based Apache server in location:

var/www/mySite

I would place robots.txt in that directory (alongside index.php) containing this:

User-agent: *
Disallow: /

right?

Does that stop robots indexing the whole server or just the site in var/www/mySite? For example would the site in var/www/myOtherSite also have robots blocked? Because I just want to do it for the one site.

Thanks!

Adam Waite
  • 19,175
  • 22
  • 126
  • 148

1 Answers1

2

Robots (well-behaved robots, that is -- honouring robots.txt is entirely voluntary) will use the robots.txt found in the root of your domain. If mySite is served off mysite.com and myOtherSite is served off myothersite.com, then your robots.txt would only be served on mysite.com and this works as intended.

To test, just head to http://myothersite.com/robots.txt and verify that you get a 404.

Thomas
  • 174,939
  • 50
  • 355
  • 478