robots.txt allow all except few sub-directories

Question

I want my site to be indexed in search engines except few sub-directories. Following are my robots.txt settings:

robots.txt in the root directory

User-agent: *
Allow: /

Separate robots.txt in the sub-directory (to be excluded)

User-agent: *
Disallow: /

Is it the correct way or the root directory rule will override the sub-directory rule?

score 15 · Accepted Answer · edited May 23 '17 at 12:33

15

No, this is wrong.

You can’t have a robots.txt in a sub-directory. Your robots.txt must be placed in the document root of your host.

If you want to disallow crawling of URLs whose paths begin with /foo, use this record in your robots.txt (http://example.com/robots.txt):

User-agent: *
Disallow: /foo

This allows crawling everything (so there is no need for Allow) except URLs like

http://example.com/foo
http://example.com/foo/
http://example.com/foo.html
http://example.com/foobar
http://example.com/foo/bar
…

edited May 23 '17 at 12:33

Community

1
1

answered Feb 15 '15 at 13:14

unor

92,415
26
211
360

Can I have a separate robots.txt for sub-domain directory? – Kunwarbir S. Feb 17 '15 at 11:09
1

Each subdomain is a new *host*, so for `foo.example.com`, the robots.txt needs to be available at `foo.example.com/robots.txt`. How you achieve this on the server-side doesn’t matter as long as bots see the robots.txt when they visit this reserved URL. – unor Feb 17 '15 at 13:32

score 1 · Answer 2 · edited Feb 13 '15 at 10:01

1

Yes there are

User-agent: *
Disallow: /

The above directive is useful if you are developing a new website and do not want search engines to index your incomplete website. also,you can get advanced infos right here

edited Feb 13 '15 at 10:01

ksokol

8,035
3
43
56

answered Feb 13 '15 at 09:36

Ganga

43
1

4

This would block everything which is not what the OP wanted. – Mar 13 '15 at 07:11
3

Also note that not all crawlers honor robots.txt, so it's not a replacement for real access control mechanisms. – Robert Nov 10 '17 at 15:51

score 0 · Answer 3 · answered Feb 13 '15 at 12:50

0

You can manage them with robots.txt which sits in the root directory. Make sure to have allow patterns before your disallow patterns.

answered Feb 13 '15 at 12:50

minion

4,313
2
20
35

robots.txt allow all except few sub-directories

3 Answers3

Linked