Robots.txt exclusion pattern

Question

I'm looking to ignore all URLs from bingbot that contain a query string variable at1= but i'm not clear on the pattern that should be used.

Should one of these work?

User-agent: bingbot
Disallow: /*at1=
Disallow: *at1=*

To confirm, I could have a URL like this:

/some-page?at1=4&ca1=5&op1=8

I still want /some-page to get visited but not when the URL listed has a query string parameters

EDIT

A poster has actually suggested a PHP solution that might make sense in terms of fine grained targeting but i'm not sure what the correct HTTP Status code would be to send to tell the spider it's not welcome. Any ideas?

score 1 · Answer 1 · answered Sep 16 '12 at 18:32

1

if ( strstr($_SERVER['HTTP_USER_AGENT'], "www.bing.com/bingbot.htm") and isset($_GET['at1']) ) :
        @header("HTTP/1.0 480 Temporarily Unavailable");
        die;
endif;

answered Sep 16 '12 at 18:32

user956584

5,316
3
40
50

This might make the most sense actually - any reason you chose 480 specifically? – robjmills Sep 17 '12 at 08:43
Realy not remember i think maybe i use 408 because some category in my script whose unalienable so temporary i send 408. You can use 404 – user956584 Sep 17 '12 at 10:30
you've posted 480 not 408 though? – robjmills Sep 17 '12 at 13:08
sorry error in comment ;) 410 Gone [ is better ] -> Indicates that the resource requested is no longer available and will not be available again. choose something from 4XX – user956584 Sep 17 '12 at 14:33
1

thinking that 403 Forbidden is probably best actually – robjmills Sep 17 '12 at 15:23

Robots.txt exclusion pattern

1 Answers1