Dynamic robots.txt based on hostname

Question

Is there a way to swap out a robots.txt file in nginx based on hostname? I currently have www.domain.com and backup.domain.com pointing at the same nginx server, but I don't want Google indexing backup.domain.com.

score 2 · Accepted Answer · answered May 01 '13 at 20:32

2

Two options:

An if statement based on $http_host that rewrites to one of two text files on the backend.
Have robots.txt be rewritten to a dynamic script (robots.php etc.) that responds accordingly.

answered May 01 '13 at 20:32

ceejayoz

32,910
7
82
106

score 1 · Answer 2 · answered Mar 25 '19 at 17:21

There is a proper way with map

map $http_host $examplecom_robotstxt {
  hostnames;
  default robotstxt/development.txt;
  example.com robotstxt/production.txt;
}

server {
  ...
  location = /robots.txt {
    rewrite .* /$examplecom_robotstxt break;
  }
  ...
}

Please note that map definition should be outside of server definition.

score 0 · Answer 3 · answered May 01 '13 at 23:09

0

For reference, this is the syntax I used:

if ($http_host != "www.domain.com") {
    rewrite ^/robots.txt /robots-backup.txt last;
}

This was inside the location block

answered May 01 '13 at 23:09

Noodles

1,386
3
18
29

`if` in `location` is evil. don't use it. never. – cadmi May 02 '13 at 04:02
Can you suggest a better solution? – Noodles May 02 '13 at 09:35
two `server {}` sections, of course. it's right way, recommended by nginx author. – cadmi May 05 '13 at 08:27

Dynamic robots.txt based on hostname

3 Answers3