0

We develop websites and we host the QA environment on the same server as the production environment. I want to serve a specific robots.txt for all QA sites but not for the production sites. We have a lot of sites so I do not want to manyallu update hundreds of vhost configuration blocks. The QA sites are easily identifyable from both the domain name and the directory they are in. QA and production sites are grouped in two different directories. All QA sites are hosted under *.qa.mycompany.com.

Example production: host: example.org docroot: /var/www/production/example.org

Example QA: host: example.qa.mycompany.com docroot: /var/www/qa/example.org

Is there any way to configure Apache to serve a robots.txt for all QA sites but not the production sites without having to update all QA vhost configs?

Sander Marechal
  • 289
  • 4
  • 11

2 Answers2

1

You could add this

<Directory "/var/www/qa/*">
  Redirect permanent robots.txt http://example.com/qa_robots.txt
</Directory>

Than you could put your qa-robots.txt on a public areay on a global host, the Directory-block should to a redirect to it for all /qa/ folders.

Search engines should also follow this redirection.

Instead of redirection you can also add

ErrorDocument 404 "/var/www/qa_robots.txt"

But this will send an 404 errorcode I´m not sure how searchbots reackt on this. I think there could also be a way to do this with mod_rewrite and a matching RewriteCond

Radon8472
  • 185
  • 8
  • I read that many robots do not follow redirects when fetching robots.txt, but I'm going to try this with a RewriteRule in the directory block. – Sander Marechal Oct 13 '15 at 12:02
1

You can do it with mod_alias & If statement, add this to your server conf : 

Alias /robots.txt /path/to/robots.txt
Alias /qa_robots.txt /path/to/robots.txt

<If "%{DOCUMENT_ROOT} =~ /(\x2Fvar\x2Fwww\x2Fqa\x2F)/">
RewriteRule ^robots\.txt$  qa_robots.txt
</If>

If statement is only available on Apache 2.4, and you tagged your question as 2.2 & 2.4 ... so it may not solve your trouble if you are using Apache 2.2

As you can't escap / in if statement, / are replaced by x2F

I didn't tested it, but it should theoricaly works.

Froggiz
  • 3,043
  • 1
  • 19
  • 30