0

I have many pages whose links are as follow:

http://site.com/school_flower/
http://site.com/school_rose/
http://site.com/school_pink/

etc. I can't block them manually. How could i block these kind of pages, while i have hundreds fo links of above type and not wanted to write each line for each link.

user2170554
  • 11
  • 1
  • http://www.robotstxt.org/robotstxt.html – rath Jun 07 '13 at 10:32
  • there is a prefix for every link i.e, school_ and hundreds of links with that prefix. is there any way to block with one line? – user2170554 Jun 07 '13 at 10:35
  • will this work? User-agent: * Disallow: /school_*/ – user2170554 Jun 07 '13 at 10:37
  • Do try it but if I remember correctly, it won't. I tried to do a similar thing once and I found out robots.txt don't accept wildcards. Also [take a look at this question](http://stackoverflow.com/questions/4833413/robots-txt-is-this-wildcard-rule-valid) – rath Jun 07 '13 at 10:56

2 Answers2

0

You can't.

robots.txt is a very simple format. But you can create a tool that will generate that file for you. That should be fairly easy, if you have a list of URLs to be blocked, one per line, you just have to prepend Disallow: to each line.

That said, the fact that you want to block many urls is an alarm. Probably, you are doing something wrong. You could ask a question about your ultimate goal and we would give you a better solution.

kirelagin
  • 13,248
  • 2
  • 42
  • 57
0

Continuing from my comment:

user-agent: *
Disallow: /folder/

Of course you'll have to place all files you don't want robots to access under a single directory, unless you block the entire site by Disallow: /

In responce to your comment, kirelagin has provided the correct answer.

rath
  • 3,655
  • 1
  • 40
  • 53