I have website
Main domain www.domain.com
Sub Domains
- www.movie.tollywood.domain.com www.movie.hollywood.domain.com
www.songs.hollywood.domain.com
I don't want to crawl and index the subdomains in all search engine.
Asked
Active
Viewed 38 times
-1
-
http://www.robotstxt.org/ – hoss Aug 06 '15 at 17:53
-
I Know the robots.txt but how to restrict the crawl & Index. Useragent * Diallow: / This is correct or Wrong – Rakesh Aug 07 '15 at 02:15
1 Answers
0
You will need to create a separate robots.txt file for each sub domain which will go into the root of those sub domains.
And to stop ALL search engines from crawling them you would add the following to each of them...
User-agent: *
Disallow: /
The above example is a wildcard as noted by the *, this means it targets all search engines.
If you only want to target Googlebot then you would add the following...
User-agent: Googlebot
Disallow: /
Billy Bragg in the UK
- 105
- 1
- 11