Stack Overflow
Stack Overflow

Questions tagged [noindex]

The noindex value of an HTML robots meta tag requests that automated Internet bots avoid indexing a web page.

The noindex value of an HTML robots meta tag requests that automated Internet bots avoid indexing a web page.

Example:

<html>
<head>
 <meta name="robots" content="noindex" />
 <title>Don't index this page</title>
</head>
128 questions
3
votes
1 answer

Robot.txt noindex

I have 2 web sites. First one is "test-www.xxxxxxx.net" and another one is "www.xxxxx.net". I want to make a dynamic no-index meta tag. Google robots can index my live site but I dont want it for my test site. Normaly I may add an attribute and make…
doganilker
  • 103
  • 2
  • 16
3
votes
2 answers

How to no index specific URLS?

I was searching around on how to no index specific URLs but I havent found any specific info on the following. By adding the below I would be able to no…
jiko
  • 39
  • 1
  • 2
3
votes
2 answers

robots.txt : how to disallow subfolders of dynamic folder

I have urls like these: /products/:product_id/deals/new /products/:product_id/deals/index I'd like to disallow the "deals" folder in my robots.txt file. [Edit] I'd like to disallow this folder for Google, Yahoo and Bing Bots. Does anyone know if…
Kulgar
  • 1,855
  • 19
  • 26
2
votes
1 answer

How to set noindex for Google only?

These days Google has this tag I can use to tell it not to index a page My question is do I need both the generic index and the Google noindex tags and does the order matter? Suppose I want everyone else…
Patrioticcow
  • 26,422
  • 75
  • 217
  • 337
2
votes
3 answers

Django: At the view level, can I add a 'noindex' header to a 'redirect' response?

I can use robots.txt, and I can use custom Django middleware, but I'd like to know whether this is something that can be done at the view level.
Christian Brink
  • 663
  • 4
  • 18
2
votes
3 answers

Ways to keep Google from indexing Sites/Content

I've a case on my Hand where I must be super duper sure that google (or any yahoo / bing for that matter) does not index specific content, so the more redundant, the better. As far as i know there are 3 Ways to accomplish that, I wonder if there are…
Hannes
  • 8,147
  • 4
  • 33
  • 51
2
votes
2 answers

Java jsoup html parsing robot index / bot detection, noindex

Firstly, this is not a duplicated question, because I have already checked almost all 503 / robot index problems. None of them solved my problem. I am trying to get giveaway list from indiegala.com but this site has some kind of protection to…
david_caruso
  • 23
  • 8
2
votes
1 answer

How to prevent search engines from indexing a span of text?

From the information I have been able to find so far, is supposed to achieve this, making a single section of a page hidden from search engine spiders. But then it also seems this is not obeyed by many browsers - so if that is the case,…
Mentalist
  • 1,530
  • 1
  • 18
  • 32
2
votes
2 answers

Do noindex pages need meta description and keywords tags?

When I specify a page as having , are the description and keywords meta tags looked at by search crawlers? Can I exclude them entirely?
tom_nb_ny
  • 110
  • 10
2
votes
2 answers

Htaccess no-follow no-index based on domain ending

I am trying to make our htaccess file as dynamic as possible, and wish to allow indexing of our .com site, but not of our .info site. We are using the same htaccess on both servers. Is there an easy way to conditionally decide this in…
ljrh
  • 439
  • 7
  • 20
2
votes
3 answers

Stop indexing certain images

So I have a robots.txt file which contains: User-agent: Googlebot-Image Disallow: / At most, it stops the indexing of a lot of images. However, I also have affiliate images that when you search come up in results in Google (and assumably other…
markb
  • 1,100
  • 1
  • 15
  • 40
1
vote
0 answers

Noindex 4000 urls using x-tag-robots via htaccess

I need to noindex 4000 URLs. I wanna do it using x-robots-tag via htaccess, but I don't want to overload the htaccess with such infinite list of URLs, so... The idea would be to create the list in a separate file (noindexurls.txt) and then using a…
Bobby
  • 23
  • 4
1
vote
1 answer

Using X robot tags to in .htaccess file to de index query strings URL from Google

I am looking for a solution to deindex all the URLs with query strings ?te= from Google. From example I want to deindex all the URLs https://example.com/?te= from Google. Google has currently indexed 21k URLs with the same query string and I want…
mehtab fatima
  • 11
  • 1
1
vote
1 answer

How to no-index specific pages on shopify? Don't want to use "contains" example for shopify, can I just type "="? And what denotes as the handle?

Shopify says to utilize the following sample to noindex a specific page in shopify. To exclude a specific page, paste the following code in the section: {% if handle contains 'page-handle-you-want-to-exclude' %}
KaraSEO
  • 11
  • 2
1
vote
0 answers

Website page "Excluded by ‘noindex’ tag" but tag is not in the code

After buying hosting and domain for my site, I uploaded the content. Then I read about how Google finds and index websites and pages, I went to the Google Search Console and registered my website property successfully. But in the "pages" section I…
Lorenzo Longhi
  • 11
  • 1
1
2
3
…
8 9