-1

I have a rails application. In my comments section I have report as offensive content' link on some (article) pages of the site. I want to get crowed the articles by search engines but not particular link report as offensive content.

So , In brief , My question is - How to Prevent spiders/search engines from following the 'report as offensive content' link of my site ?

sunnyrjuneja
  • 6,033
  • 2
  • 32
  • 51
Manish Shrivastava
  • 30,617
  • 13
  • 97
  • 101

2 Answers2

1

You can specify what links for web crawlers not to crawl with a robots.txt file.

Learn more here:

http://www.robotstxt.org/

Here's an example. Create a robots.txt in the most top level directory of your application so it can be accessed by your domain.com/robots.txt.

User-agent: *
Disallow: /offensive

This says for every User-Agent, ignore the /offensive url.

sunnyrjuneja
  • 6,033
  • 2
  • 32
  • 51
1

I found one answer

Pros : - Its easy . Just One line

<a href="mypage.html" rel="nofollow" />

But Cons :-

Our tests show that some search engines do crawl and index nofollow links. The nofollow tag will probably diminish the ranking value a link will provide but it cannot be reliably used to stop search engines from following a link.

Manish Shrivastava
  • 30,617
  • 13
  • 97
  • 101
  • But as per this blog it says nofollow supports many search engines http://antezeta.com/news/avoid-search-engine-indexing#SEI20 – Manish Shrivastava Oct 11 '12 at 07:47
  • 1
    there are a LOT of things that you can do to identify a web-crawler and prevent it from crawling a specific page. See my answer for more details: http://stackoverflow.com/questions/8404775/how-to-identify-web-crawler/8405803#8405803 – Kiril Oct 12 '12 at 16:32