robots.txt file with more restrictive rules for certain user agents

Question

I'm a bit vague on the precise syntax of robots.txt, but what I'm trying to achieve is:

Tell all user agents not to crawl certain pages
Tell certain user agents not to crawl anything

(basically, some pages with enormous amounts of data should never be crawled; and some voracious but useless search engines, e.g. Cuil, should never crawl anything)

If I do something like this:

User-agent: *
Disallow: /path/page1.aspx
Disallow: /path/page2.aspx
Disallow: /path/page3.aspx

User-agent: twiceler
Disallow: /

..will it flow through as expected, with all user agents matching the first rule and skipping page1, page2 and page3; and twiceler matching the second rule and skipping everything?

score 1 · Answer 1 · answered Jun 08 '10 at 06:22

1

It would appear that you have a better understanding than you realize. :)

answered Jun 08 '10 at 06:22

Justin Scott

8,798
1
28
39

So that all looks good to you? It will use the first rule everything and the second rule for twiceler? – Carson63000 Jun 08 '10 at 06:35

score 0 · Answer 2 · answered Jun 09 '10 at 15:46

0

Hmm, depends on the crawler and whether it just goes on first match basis. IE twiceler might see the wildcard entry first and not check any further, so would not see the Disallow: /

answered Jun 09 '10 at 15:46

user45348

169
3

Would it be recommended to put the wildcard User-agent entry _last_ in the robots.txt file, then? – Carson63000 Jun 11 '10 at 04:48
Think og the wildcard as a catchall rule, so I always put it last. Have the specific ones first. – user45348 Jun 11 '10 at 10:55

robots.txt file with more restrictive rules for certain user agents

2 Answers2