1

I'm attempting to prevent my website from being indexed/listed:

I've set in .htaccess:

Header set X-Robots-Tag "noindex, nofollow"

Is it also necessary to set robots.txt?

User-agent: *
Disallow: /

Is there any harm in adding the robots.txt?

kylex
  • 14,178
  • 33
  • 114
  • 175

1 Answers1

0

Found the answer, if have X-Robots-Tag set, I must remove any crawl disallows in robots.txt:

Robots meta tags and X-Robots-Tag HTTP headers are discovered when a URL is crawled. If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving directives will not be found and will therefore be ignored. If indexing or serving directives must be followed, the URLs containing those directives cannot be disallowed from crawling.

via: https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag#combining-crawling-with-indexing--serving-directives

kylex
  • 14,178
  • 33
  • 114
  • 175