How to disallow specific pages in robots.txt, but allow everything else?

Question

Is this the way to do it?

User-agent: *
Allow: /
Disallow: /a/*

I have pages like:

mydomaink.com/a/123/group/4
mydomaink.com/a/xyz/network/google/group/1

I don't want to allow them to appear on Google.

Answer 1

Your robots.txt looks correct. You can test in in your Google's Webmaster Tools account if you want to be 100% sure.

FYI, blocking pages in robots.txt doe snot guarantee they will not show up in the search results. It only prevents search engines from crawling those pages. They can still list them if they want to. To prevent a page from being indexed and listed you need to use the x-robots-tag HTTP header.

If you use Apache you can place a file in your /a/ directory with the following line to effectively block those pages:

<IfModule mod_headers.c>
    Header set X-Robots-Tag: "noindex"
</IfModule>