How Disallow a mirror site (on sub-domain) using robots.txt?

Question

I have a website saying :

http://domain.com/

mirror site on

http://cdn.domain.com/

I don't want cdn to be indexed. How can I write robots.txt rule to avoid the cdn from being indexed without disturbing my present robots.txt excludes.

My present robots.txt excludes :

User-agent: *
Disallow: /abc.php

How can I avoid cdn.domain.com from being indexed ?

User-agent: *
Disallow: /abc.php

`Try this`: I guess you could put a seperate robots.txt file inside the sub-domain and do your exclusions from there. You can block an entire subdomain via robots.txt, however you'll need to create a robots.txt file and place it in the root of the subdomain, then add the code to direct the bots to stay away from the entire subdomain's content. `User-agent: *` - `Disallow: /` — Funk Forty Niner, Jun 05 '13 at 17:45
Consult http://stackoverflow.com/questions/4904642/disallow-or-noindex-on-subdomain-with-robots-txt — Funk Forty Niner, Jun 05 '13 at 17:47
you could make a robots-cdn.txt and indescriminately rewrite robots.txt to that from the subdomain's virtural host. — Orangepill, Jun 05 '13 at 17:48
Can-not have different `robots.txt` since its a cdn and can-not have different codebase. — Yugal Jindle, Jun 05 '13 at 17:49
I'm not suggesting that ... I'm suggesting an additional robots file called robots-cdn.txt that will, by the magic of rewriting be served up on the cdn instead of the robots.txt — Orangepill, Jun 05 '13 at 17:54
It's basically the non-accepted answer on the question that @Fred referenced — Orangepill, Jun 05 '13 at 17:56
@Orangepill I'm almost afraid to `post it as an answer` lol! We don't know what the OP tried, or hasn't tried. — Funk Forty Niner, Jun 05 '13 at 17:57
Thanks @Orangepill. Guys thanks for a great effort. All those who mis-understood the questions voted it down. — Yugal Jindle, Jun 06 '13 at 09:39

score 10 · Accepted Answer · edited Jun 06 '13 at 09:46

10

in your root .htaccess file add the following

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^Amazon.CloudFront$
RewriteRule ^robots\.txt$ robots-cdn.txt

And then create a separate robots-cdn.txt:

User-agent: *
Disallow: /

When accessed through via http://cdn.domain.com/robots.txt will return the contents of the robots-cdn.txt file... otherwise the rewrite won't kick in and the true robots.txt will kick in.

This way you are free to mirror the entire site (including the .htaccess) file with the expected behavior

Update :

HTTP_USER_AGENT did it since Amazon uses it while querying it from any location.
I have verified and it works

edited Jun 06 '13 at 09:46

Yugal Jindle

44,057
43
129
197

answered Jun 05 '13 at 19:13

Orangepill

24,500
3
42
63

Sir, I don't think this will work. Since cdn is not a machine and hence if I ask robots from cdn, it will inturn ask it from domain.com as a normal browser client. So, we will get the same robots.txt (that we don't want) – Yugal Jindle Jun 06 '13 at 07:34
http_host variable isn't referring to a network host. its referring to the host portion of a url. you can view the same variable in php an the $_SERVER[HTTP_HOST] superglobal – Orangepill Jun 06 '13 at 07:47
You were right, although `HTTP_USER_AGENT` worked instead for `amazon`. I have made the required changes in the answer. Thanks. – Yugal Jindle Jun 06 '13 at 09:24

score 0 · Answer 2 · answered Jun 05 '13 at 18:23

0

If the codebase are the same, you can generate your robots.txt dynamically and change its content depending on the requested (sub)domain.

answered Jun 05 '13 at 18:23

ZeWaren

3,978
2
20
21

Hm.. can you help with some code, since I also tried to attempt what you said. – Yugal Jindle Jun 06 '13 at 07:15
What language and platform are you using? – ZeWaren Jun 06 '13 at 08:57

How Disallow a mirror site (on sub-domain) using robots.txt?

2 Answers2