What does "Allow: /$" mean in robots.txt

Question

When digging through a Google robots.txt file I noticed a line that I was not familiar with.

What does the below code mean in the context of a robots.txt file?

Allow: /$

Does the '$' change the meaning any from simply saying

Allow: /

Answer 1

According to SearchEngineLand..

You can match the end of the string using the dollar sign ($). For example, to block URLs that end with .asp:

User-agent: Googlebot
Disallow: /*.asp$

Looks like it acts a bit like a regular expression.

Answer 2

Some robots understand pattern matching. $ is used in regular expressions to mark end of line. So this rule should Allow / but not /foo.

I have no source for robots.txt particular though.

Answer 3

Allow and $ are not part of the original robots.txt specification. Google proposed and added them to their own interpretation of robots.txt. If you use them, be aware that most bots will not know how to deal with them. Most bots will ignore the Allow directives and treat $ as a literal.

For Google, Allow should only be used as an exception to Disallow. Take this case:

User-Agent: *
Disallow: /
Allow: /$

Disallow: / means that no crawling is allowed for the entire site. By default rules are "starts with" rules and every URL path on your site starts with a slash.

Allow: /$ is an exception that means that the home page can be crawled. The $ means "ends with" and makes that rule an exact match against the URL path.

Googlebot would be allowed to crawl the home page of your site, but no other pages. Most bots that ignore Allow directives would not be able to crawl your site at all.

Using Allow: / instead, would allow crawling of the entire site. You shouldn't use that. If you want to allow your entire site to be crawled, you should instead use a robots.txt that disallows nothing:

User-Agent: *
Disallow: