-2

I have a question regarding robots.txt file.

Disallow: Blog/*+*

What does that mean?

Werner Henze
  • 16,404
  • 12
  • 44
  • 69

1 Answers1

2

In theory it would stop a robot that chose to respect it from accessing any part of the website that began with Blog/+ ; however, the bot doesn't have to respect it, and since it isn't starting with a directory indicating slash there is no telling how people's robots will deal with it.

from : http://www.robotstxt.org/orig.html

Disallow The value of this field specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved. For example, Disallow: /help disallows both /help.html and /help/index.html, whereas Disallow: /help/ would disallow /help/index.html but allow /help.html. Any empty value, indicates that all URLs can be retrieved. At least one Disallow field needs to be present in a record.

Semicolons and Duct Tape
  • 2,927
  • 4
  • 20
  • 35