0

For example, I want to block /foo.php, /foo/foo.php and every other similar URL in robots.txt, only leaving /, /foo/, etc. behind.

In other words, I want to block everything except the directories.

How is this possible?

Lucas
  • 16,930
  • 31
  • 110
  • 182

1 Answers1

1

If all URLs you want to block end in (or contain) .php, you could use

User-agent: *
Disallow: /*.php

However, this uses the * wildcard, which maybe not all parsers support (Google supports it).

unor
  • 92,415
  • 26
  • 211
  • 360