how do I use wildcards in robots.txt?

Question

I have seen lots of questions sort of like this, but not exactly.

In my robots.txt I want to do something like:

Disallow: /home/[anything]/[anything]/personal/

yet allow access to /home, home/xxx, /home/xxx/yyy/, etc

where * can be any intervening directory. The example above is what I've tried and lots of variations on this, nothing seems to work.

*/personal

/home*/personal

just about every permutation.

Can this be done with the limited wildcards in robots? If so -- how?

There's lots of info out there on robots.txt, and a lot of it conflicting!]

robots.txt doesn't allow or disallow access, it is simply a set of rules you would *like* robots to follow based on their user-agent strings or other criteria. For access control, you want to use something like .htaccess — David Wilkins, Dec 24 '13 at 20:21
possible duplicate of [robots.txt : how to disallow subfolders of dynamic folder](http://stackoverflow.com/questions/17068876/robots-txt-how-to-disallow-subfolders-of-dynamic-folder) — Justin Wood, Dec 24 '13 at 20:23
Understood. I guess I wasn't clear enough that search robots are what I'm targeting. — Bill, Dec 24 '13 at 20:50

score 1 · Answer 1 · answered Dec 24 '13 at 20:28

1

The Google crawler understands patterns within robots.txt, but keep in mind not all bots do (or will even follow robots.txt at all for that matter). For example:

Disallow: /*foo.html$

Would disallow any foo.html document.

Check out the Robots.txt specification for more information. They also have a test tool to verify if your rules are working correctly.

answered Dec 24 '13 at 20:28

Mike Christensen

88,082
50
208
326

So, in my case I should try: */statistics$ for example? Thanks. – Bill Dec 24 '13 at 20:43
Yea sounds right. Use the test tool as well to verify things are working correctly, and keep in mind it probably won't block every search engine. – Mike Christensen Dec 24 '13 at 20:45
Understood. I'm just worried about the big one! – Bill Dec 24 '13 at 20:47

score 1 · Answer 2 · answered Apr 03 '14 at 16:26

Google Webmaster Tools has a great tool for trying out robots.txt directives (at least as Google Bot will interpret them). You can add a line into the robots file right inside webmaster tools, and add a url you want to make sure gets blocked, hit the test button and it will tell you if it's blocked or allowed and if blocked, which directive (line number) did it. See attached screenshot. (ah booo to stackoverflow - it won't allow me to add the screenshot, cause I don't have any 'reputations').

how do I use wildcards in robots.txt?

2 Answers2