Robots.txt file and Googlebot crawability

Question

Will this robots.txt allow Googlebot to crawl my site or not?

Disallow: /
User-agent: Robozilla
Disallow: /
User-agent: *
Disallow:
Disallow: /cgi-bin/
Sitemap: https://koyal.pk/sitemap/sitemap.xml

Answer 1

If you want to know how Google will react to a robots.txt file, you should get an official answer by testing in Google's robots.txt testing tool. Here I have the results of such a test using the robots.txt that you provided:

Googlebot will be able to crawl the site, however Google tells you that the robots.txt syntax you are using as a problem. I see several problems:

• A Disallow directive must have a User-agent directive somewhere above it.
• There should be a new line before each User-agent directive (except the one at the beginning of the file.)
• The Disallow: line means "allow all crawling". That should only be used if there are no other Disallow rules.

A syntactically correct robots.txt that I think would do what you intend is:

User-agent: Robozilla
Disallow: /

User-agent: *
Disallow: /cgi-bin/
Sitemap: https://koyal.pk/sitemap/sitemap.xml

That would provent the Robozilla bot from crawling while allowing all other bots (including Googlebot) to crawl everything except the /cgi-bin/ directory.