Robots.txt For Two Separate Websites That Share the Same Root

Question

I have to make a robots.txt file for two different websites but they are both on the same root directory.

They are as different as google.com and yahoo.com but they are both tied to the same place so one change effects both. Looking for a solution to create a single robots file for both domains, any help would be greatly appreciated.

Found this other resource but I am not sure if it will apply: Robots.txt for multiple domains

It will work if you can use apache mod rewrite. If this is a dynamic site, you could also route the request for robots.txt to robots.jsp (for example). — Kevin Seifert, Mar 11 '15 at 14:29
The link you posted will work then (creating the .htaccess file) — Kevin Seifert, Mar 12 '15 at 14:43

score 0 · Answer 1 · answered Sep 27 '16 at 07:53

The solution given in the link that you have mentioned will definitely work. I am using the same setup on nginx server. As an alternative I though I could also try to add the domain to the robots.txt file.Like:

Disallow: domain1.com/path/to/disallow 
Disallow: domain2.com/path/to/disallow

But I am not 100% sure if this will work.

Robots.txt For Two Separate Websites That Share the Same Root

1 Answers1