robots.txt block bots crawling subdirectory

Question

I'd like to block all bots from crawling a sub directory http://www.mysite.com/admin plus any files and folders in that directory. For example there may be further directories inside /admin such as http://www.mysite.com/admin/assets/img

I'm not sure what is the exact correct declarations to include in robots.txt to do this.

Should it be:

User-agent: *
Disallow: /admin/

Or:

User-agent: *
Disallow: /admin/*

Or:

User-agent: *
Disallow: /admin/
Disallow: /admin/*

I'd suggest you never tell people where you don't want them to look ;) coming back with the answer after having checked. I'd say the first one though — tattvamasi, Sep 01 '13 at 20:33
This question appears to be off-topic because it is about SEO — John Conde, Sep 02 '13 at 01:15
@JohnConde what are you talking about? My question is specifically asking what is the correct code to use. It's a coding question. — CaribouCode, Sep 02 '13 at 16:34
Writing a robots.txt file is not coding by anyone's standards including stack overflow. — John Conde, Sep 02 '13 at 16:37

tattvamasi · Accepted Answer · 2013-09-02T13:21:21.357

0

Based on information available on the net (I can't retrieve it all but some forums actually report the problem, as in here and here, for example) I'd follow those who suggest we never tell people or bots (or both) where is that we don't want them to look ("admin" looks like sensitive content...).

After having checked, I'd confirm it's the first one you say. Reference here

edited Sep 02 '13 at 13:21

answered Sep 01 '13 at 20:37

tattvamasi

845
1
7
14

How else would I stop my admin pages or other directories appearing on google after they get crawled? – CaribouCode Sep 01 '13 at 20:42
that's a good question... I let it be. EDIT: after Google crawls them it's a bit difficult to undo I think, but I use their webmaster tools. Also, the sitemap helps a lot – tattvamasi Sep 01 '13 at 20:55
Robots.txt is there for a reason. For example if I have a directory /test and as a web designer I put all my test sites in there, I don't want them appearing on search engines before they go live. You shouldn't have to mess around with sitemaps and webmaster tools to do the same thing, especially if that content changes everyday. For that reason robots.txt. For me, it's not usually to do with sensitive content, and more about not wanting the general public to have access to those areas through search engines. – CaribouCode Sep 02 '13 at 12:05
yeah I'm telling you what I'm told...Anyhow, if the answer is the one you were looking for, kindly click "accept" and thanks! ;) – tattvamasi Sep 02 '13 at 12:10
I'll accept your response as it answers my initial question. However in my option, your answer also contains advice that is not neccessarily good, nor backed up with evidence. – CaribouCode Sep 02 '13 at 12:50
thanks. I am sure I read the advice in more than one place. I'll retrieve the link and post it in the edited answer. – tattvamasi Sep 02 '13 at 12:56

robots.txt block bots crawling subdirectory

1 Answers1