robot.txt to block directory showing

Question

Few questions

• How can you effectively block directories and their contents using robots.txt?

Is it ok to do:

User-agent: * Disallow: /group Disallow: /home

Do you have to put a trailing slash, for example:

User-agent: * Disallow: /group/ Disallow: /home/

Also what is the difference between Disallow in robots.txt and adding ?

If I want google not to show specific pages and folders in a directory, what should I do?

Answer 1

Is it ok to do:

User-agent: * Disallow: /group Disallow: /home

1. You must place these on separate lines
2. It is highly recommended that you put a trailing slash if you are trying to exlude the directories home and group

I would do something like this:

User-agent: *
Disallow: /group/
Disallow: /home/

About the trailing slash, yes, you should add it according to http://www.thesitewizard.com/archive/robotstxt.shtml:

Remember to add the trailing slash ("/") if you are indicating a directory. If you simply add

User-agent: *
Disallow: /privatedata

the robots will be disallowed from accessing privatedata.html as well as ?privatedataandstuff.html as well as the directory tree beginning from /privatedata/ (and so on). In other words, there is an implied wildcard character following whatever you list in the Disallow line.

If you do not want google to show specific pages or directories, add a Disallow line for each of these pages or directories.