18

Today whilst improving my web crawler to support the robots.txt standard, I came across the following code at http://www.w3schools.com/robots.txt

User-agent: Mediapartners-Google 
Disallow:

Is this syntax correct? Shouldn't it be Disallow: / or Allow: / depending on the intended purpose?

unor
  • 92,415
  • 26
  • 211
  • 360
dangee1705
  • 3,445
  • 1
  • 21
  • 40

1 Answers1

29
Disallow:

Will allow everything, as will:

Allow: /

You're either disallowing nothing, or allowing everything.

Ralph King
  • 994
  • 1
  • 9
  • 19
  • Thank you, that makes everything much clearer now :) – dangee1705 Apr 04 '16 at 15:00
  • 1
    WARNING: I added a robots.txt containing `User-agent: * Disallow:` then checked Google Search console 2 weeks later. The very day same day the robots.txt was created, Google stripped the site from its index. This file is almost useless unless you truly want to disallow a search engine. Don't add it just to make some SEO tool happy! – Aaron Cicali Aug 03 '17 at 23:17
  • What about empty `Allow`, does that mean allow nothing? – endo64 Mar 06 '19 at 07:19
  • `Allow:` is Google specific. Tell google it can access a page or subfolder even though its parent page or subfolder may be disallowed. – Ralph King Mar 18 '19 at 15:54