8

my client has a load of pages which they dont want indexed by google - they are all called 

http://example.com/page-xxx

so they are /page-123 or /page-2 or /page-25 etc

Is there a way to stop google indexing any page that starts with /page-xxx using robots.txt

would something ike this work?

Disallow: /page-*

Thanks

JorgeLuisBorges
  • 528
  • 3
  • 8
  • 21

3 Answers3

15

In the first place, a line that says Disallow: /post-* isn't going to do anything to prevent crawling of pages of the form "/page-xxx". Did you mean to put "page" in your Disallow line, rather than "post"?

Disallow says, in essence, "disallow urls that start with this text". So your example line will disallow any url that starts with "/post-". (That is, the file is in the root directory and its name starts with "post-".) The asterisk in this case is superfluous, as it's implied.

Your question is unclear as to where the pages are. If they're all in the root directory, then a simple Disallow: /page- will work. If they're scattered across directories in many different places, then things are a bit more difficult.

As @user728345 pointed out, the easiest way (from a robots.txt standpoint) to handle this is to gather all of the pages you don't want crawled into one directory, and disallow access to that. But I understand if you can't move all those pages.

For Googlebot specifically, and other bots that support the same wildcard semantics (there are a surprising number of them, including mine), the following should work:

Disallow: /*page-

That will match anything that contains "page-" anywhere. However, that will also block something like "/test/thispage-123.html". If you want to prevent that, then I think (I'm not sure, as I haven't tried it) that this will work:

Disallow: */page-

Jim Mischel
  • 131,090
  • 20
  • 188
  • 351
  • Hi - yep the page/post thing was just a typo. This is in the distant past so cant check your solution, but I just got a notification that people are still looking at this, so I thought I'd correct the typo. If anyone can confirm Jim's solution I can mark as correct. – JorgeLuisBorges Oct 27 '17 at 16:21
1

It looks like the * will work as a Google wild card, so your answer will keep Google from crawling, however wildcards are not supported by other spiders. You can search google for robot.txt wildcards for more info. I would see http://seogadget.co.uk/wildcards-in-robots-txt/ for more information.

Then I pulled this from Google's documentation:

Pattern matching

Googlebot (but not all search engines) respects some pattern matching.

To match a sequence of characters, use an asterisk (*). For instance, to block access to all >subdirectories that begin with private:

User-agent: Googlebot Disallow: /private*/

To block access to all URLs that include a question mark (?) (more specifically, any URL that begins with your domain name, followed by any string, followed by a question mark, followed by any string):

User-agent: Googlebot Disallow: /*?

To specify matching the end of a URL, use $. For instance, to block any URLs that end with .xls:

User-agent: Googlebot Disallow: /*.xls$

You can use this pattern matching in combination with the Allow directive. For instance, if a ? indicates a session ID, you may want to exclude all URLs that contain them to ensure Googlebot doesn't crawl duplicate pages. But URLs that end with a ? may be the version of the page that you do want included. For this situation, you can set your robots.txt file as follows:

User-agent: * Allow: /?$ Disallow: /?

The Disallow: / *? directive will block any URL that includes a ? (more specifically, it will block any URL that begins with your domain name, followed by any string, followed by a question mark, followed by any string).

The Allow: /*?$ directive will allow any URL that ends in a ? (more specifically, it will allow any URL that begins with your domain name, followed by a string, followed by a ?, with no characters after the ?).

Save your robots.txt file by downloading the file or copying the contents to a text file and saving as robots.txt. Save the file to the highest-level directory of your site. The robots.txt file must reside in the root of the domain and must be named "robots.txt". A robots.txt file located in a subdirectory isn't valid, as bots only check for this file in the root of the domain. For instance, http://www.example.com/robots.txt is a valid location, but http://www.example.com/mysite/robots.txt is not.

Note: From what I read this is a Google only approach. Officially there is no Wildcard allowed in robots.txt for disallow.

Travis Pessetto
  • 3,260
  • 4
  • 27
  • 55
  • I've checked that out - doesn't seem to work. checked with [this](http://www.searchenginepromotionhelp.com/m/robots-text-tester/robots-checker.php) - and wasn;t allowed to add it in googles robots.txt builder – JorgeLuisBorges Jul 28 '11 at 13:59
  • It that doesn't work you have either the option of blocking a directory (if all of them are in there) or trying to add meta-content/link content dynamically using PHP, Javascript or another one of your choosing. – Travis Pessetto Jul 28 '11 at 14:02
  • I would suggest PHP or something similar as Google will probably will not load the Javascript. – Travis Pessetto Jul 28 '11 at 14:04
  • @JoreLuisBorges, Note Googlebot syntax will not pass the validation of the test. You must test against Googles own tools, I believe you must manually add it. You can read more here http://www.google.com/support/webmasters/bin/answer.py?answer=156449 just click on the manual part. – Travis Pessetto Jul 28 '11 at 14:18
  • seems a bit wierd that google's own webmaster tools wont validate that method! In the endi I've gone for a method that adds the robot instruction in the meta of each page through the cms. – JorgeLuisBorges Aug 01 '11 at 13:56
0

You could put all the pages that you don't want to get visited in a folder and then use disallow to tell bots not to visit pages in that folder.

Disallow: /private/

I don't know very much about robots.txt so I'm not sure how to use wildcards like that Here, it says "you cannot use wildcard patterns or regular expressions in either User-agent or Disallow lines." http://www.robotstxt.org/faq/robotstxt.html

obesechicken13
  • 795
  • 1
  • 12
  • 24