Questions tagged [x-robots-tag]

The x-robots tag is a field in the HTTP response header that allows sites to tell search engines and other crawlers whether or not they are not allowed to access the content found on the URL. In this way, it is very similar to the meta robots tag or the robots.txt file. The difference is that this information is found in the HTTP response header instead of the page source or the robots.txt file on the root of the domain.

The x-robots tag is a field in the HTTP response header that allows sites to tell search engines and other crawlers whether or not they are not allowed to access the content found on the URL. In this way, it is very similar to the meta robots tag or the robots.txt file. The difference is that this information is found in the HTTP response header instead of the page source or the robots.txt file on the root of the domain.

A typical x-robots tag:

X-Robots-Tag: noindex

Similar to the meta robots tag, the x-robots tag can contain multiple values, comma-separated.

Below are the most important values:

  • all: The default value. No restrictions for robots.
  • noindex: Exclude this URL from being indexed
  • nofollow: Do not follow links on this page
  • none: Do nothing with this page (same as "noindex, nofollow")

The x-robots tag is relatively new and not as widely used or accepted as robots.txt or the meta robots tag. Google, Bing, and Yahoo support it, but other search engines may not.

24 questions
1
vote
0 answers

Get X-robots-tag in PHP

I would like to check if a list of URL doesn't have a noindex. So I check already < meta name="robots" > with DOM, but I would like to also check the X-robots-tag. Is there an easy way to do it? The only way I can imagine is a foreach() on…
jd440
  • 47
  • 1
  • 7
1
vote
2 answers

x-robots tag for query string parameter

I want to block a query string url's in .htaccess using x-robots tag. The url's are something like: https://www.example.com/test?limit=60 https://www.example.com/test?limit=45 https://www.example.com/test?limit=all I need to block the…
chappers
  • 466
  • 1
  • 6
  • 17
1
vote
2 answers

How to set missing X-Robots-Tag for HTTP 301 response?

I want to set noindex x-robots tag for a particular bad search engine which index even redirect page, instead of final destination. At the top of my root .htaccess file, I have added below rules. Header add X-Robots-Tag…
0
votes
0 answers

sitemap blocked by 'noindex' detected in 'X-Robots-Tag' http header

I have the following picture. enter image description here I don't know when my website was like this. I checked in htcacess and in source webiste there is no noindex in header. In fact, even if I have deleted and reinstalled WP completely, my…
0
votes
0 answers

Add HTTP Response Header when domain does not start with "www."

Attempting to use URL Rewrite module in IIS 10 on a Windows 10 VM to inject/add a response header if the domain is NOT the "www." domain but it's not getting added. There are several bindings on this site and we only want pages indexed when the url…
0
votes
0 answers

Setting X-Robots-Tag header on multiple queries and pathes via htaccess

I'm trying to set X-Robots-tag for some url parameters and specific path, simulatenously from htaccess. I have google indexing urls such as https://www.shantima.com/shop/large-waffle-linen-bath-towel/?wishlist-action…
J.o.
  • 11
  • 4
0
votes
0 answers

Vue SPA robots meta tag not indexed by Google

For my Vue SPA for which SEO is important; I have written my own function for setting meta tags per route I fire a simple function like this in the created lifecycle hook of my views: document .querySelector('meta[name="robots"]') …
Rogier
  • 142
  • 11
0
votes
2 answers

How to stop google-bot from indexing a folder inside my s3 bucket?

I have an amazon s3 bucket with static website hosting setup + cloudfront. I have a folder inside the s3 bucket [ example.com/Books ] which contains pdf files. I've submitted a sitemap in google search console [ which doesn't contain any pdf urls ]…
0
votes
1 answer

Override "X-Robots-Tag: noindex" in Tumblr

Tumblr automatically inserts a "X-Robots-Tag: noindex" line in the HTTP header for all tagged pages; e.g. "blog.tumblr.com/tagged/tag". Is there a way to override this? Or is there a way to modify the header in any way using scripts or any other…
NDG
  • 41
  • 7
1
2