Can I use robots.txt to block certain URL parameters?

Question

Before you tell me 'what have you tried', and 'test this yourself', I would like to note that robots.txt updates awfully slow for ~~my site~~any site on search engines, so if you could provide theoretical experience, that would be appreciated.

For example, is it possible to allow:

http://www.example.com

And block:

http://www.example.com/?foo=foo

I'm not very sure.

Help?

you could try to use a robot emulator, but using **deny** in robots.txt doesn't mean all robots will follow it! — CSᵠ, Jan 02 '13 at 23:35

score 7 · Accepted Answer · answered Jan 02 '13 at 23:39

7

According to Wikipedia, "The robots.txt patterns are matched by simple substring comparisons" and as the GET string is a URL you should be able to just add:

Disallow: /?foo=foo

or something more fancy like

Disallow: /*?*

to disable all get strings. The asterisk is a wildcard symbol so it matches one or many characters of anything.

Example of a robots.txt with dynamic urls.

answered Jan 02 '13 at 23:39

Sean Dawson

5,587
2
27
34

Just wondering, is it possible to block all URLs with `?foo=foo`, not just the base directory? – Lucas Jan 02 '13 at 23:43
Sure if you remove the forward slash it will just match the string anywhere in the URL. – Sean Dawson Jan 03 '13 at 00:45
1

Note that the `*` wildcard is not part of the original robots.txt specification. – unor Jan 03 '13 at 12:13

Can I use robots.txt to block certain URL parameters?

1 Answers1