Disallow In-Page Url Crawls

Question

I want to disallow all the bots to crawl specific type of pages. I know this can be done via robots.txt as well as .htaccess. However, these pages are generated from the database from the user's request. I have searched the internet and could not get a good answer for doing so.

My link looks like:

http://www.my_website/some_controller/some_action/download?id=<encrypted_id>

There is a view page for the users wherein all the data that is displayed comes from the database including the kind of links that I have mentioned before. I want to hide those links from the bots and not the entire page. How can I do that?

This question is off-topic for Stack Overflow as it doesn't appear to be about programming. — AStopher, Feb 20 '15 at 18:39
@Abhimanyu Simply create a `robots.txt` file at the root of your site and in it put `User-agent: * Disallow: /`, this'll stop anything crawling your site. See [here](http://www.robotstxt.org/robotstxt.html). — AStopher, Feb 20 '15 at 18:50
@ʎǝʞuoɯɹǝqʎɔ I can't do that. I need the page to be crawled but not that link. — Abhimanyu Saharan, Feb 20 '15 at 18:52
As long as `.htaccess` and `robots.txt` are autocreated, you cannot do this (unless your application allows custom rules to be inserted). — AStopher, Feb 20 '15 at 18:53

score 2 · Answer 1 · edited Feb 20 '15 at 18:45

2

Could the page not be generated with a

<meta name="robots" content="noindex">

in the head?

edited Feb 20 '15 at 18:45

BradleyDotNET

60,462
10
96
117

answered Feb 20 '15 at 18:37

Darryl Edwards

119
1
10

@BradleyDotNET The page is being crawled but I don't want those types of links to be crawled from the page where they are being displayed – Abhimanyu Saharan Feb 20 '15 at 18:50
@Abhimanyu I was just editing out the fluff. I'm no expert on this topic :) – BradleyDotNET Feb 20 '15 at 18:51

score 0 · Answer 2 · answered Mar 14 '15 at 01:58

you cannot hide stuff from bots but make it available to other traffic, afterall how do you distinguish between a bot and regular traffic... you cant without some sort of verification like them pictures of a word you type in a box. Robots.txt does not stop bots, most bots will look at it and that will stop them out of there own choice, however that is only because they are programmed to do so. They do not have to do this and therefore if they wish can ignore robots.txt completely.

Disallow In-Page Url Crawls

2 Answers2