Using X-Robot-Tag in.htaccess - Wordpress to no index query string

Question

I have looked at other questions on here regarding using the X-Robot-Tag to noindex a specific page on .htaccess.

My question is similar, although I would like to noindex a group of pages and am unsure how to do this.

I am using wordpress and am using the layered-nav filter plugin that comes with woocommerce.

I have managed to noindex the majority of pages with the Yoast plugin however each option on the layered nav filter creates a page to display filtered options. these pages are of the form

https://www.mysite.co.uk/brand/brand1/?filter_cats1=cat1

every filter has a similar url ending in a similar fashion to ?filter_cats1=cat1

Is there a way I can filter out these URLs using X-robot-Tag? As mentioned, I have only seen ways to noindex specific pages, I am yet to find a way to noindex groups of pages with similar URLs

Answer 1

If you want to add noindex, follow to all urls using the same parameter filter_cats1, so use following rule:

<IfModule mod_headers.c>
RewriteEngine On

RewriteCond %{QUERY_STRING} (^|&)filter_cats1= [NC]
RewriteRule ^ - [E=DO_SEO_HEADER:1]

Header set X-Robots-Tag "noindex, nofollow" env=DO_SEO_HEADER
</IfModule>

Answer 2

I also faced the same problem. However, I just redirect these query strings to the relevant pages. For example:

    RewriteEngine on
<if "%{QUERY_STRING} =~ /^filter_cats1=/">
RewriteRule  .  %{REQUEST_URI}?   [R=301,L]
</if>

What this does is to strip the query string or removes everything including the ? and the remaining string that follows and redirect it to relevant url. This will redirect the url with query string or https://www.mysite.co.uk/brand/brand1/?filter_cats1=cat1 to url without the query string : https://www.mysite.co.uk/brand/brand1/ You do not need to list the entire query string in the rewrite rule either. Only the first part of the string that is after ? to = would be sufficient to create the redirect. In this case, you will not need to no index the query string. A simple 301 redirect would do. You can test the redirect after implementing it and also apply it to any query string. There are more methods to deal with query strings including :

<IfModule mod_rewrite.c>
RewriteCond %{REQUEST_URI} ^/specific/url/
RewriteCond %{QUERY_STRING} key=value
RewriteRule (.*) /path/ [R=301,L]

You can read more about redirecting or removing query strings here: https://perishablepress.com/redirect-query-string-htaccess/

Answer 3

See this answer

You should put a new .htaccess in user/uploaded/ directory. In this file you will be able to specify your .htaccess rule

Header set X-Robots-Tag: "noindex"

You don't need to use FilesMatch except if you want to target specific files.##