Robots.txt for umbraco

Question

I wonder if there is a default Robots.txt for Umbraco (7.5) websites. Right now, I created one with these items. Shall I add anything else?

User-Agent: *
Disallow: /umbraco/
Disallow: /umbraco_client/
Disallow: /css
Disallow: /Errorpages/
Disallow: /scripts/
Disallow: /Config/

Dont forget your `Sitemap: /sitemap.xml` too (provided there is a sitemap). — Anth12, Sep 14 '16 at 10:06

score 5 · Answer 1 · answered Sep 13 '16 at 13:14

5

You should consider removing /css and /scripts from the list, actually. I believe Google finds it friendlier/better for those to be available.

Otherwise, maybe put /install/ in there as well, just as a precaution.

answered Sep 13 '16 at 13:14

Jannik Anker

3,320
1
13
21

Thanks @jannikAnker for helping I've also added some more :) – Ashkan S Sep 15 '16 at 09:25

score 3 · Accepted Answer · answered Sep 17 '16 at 20:18

Regarding CSS & JSS check this: https://yoast.com/dont-block-your-css-and-js-files/. I decided to cut them off lately.

Mine, always used with Umbraco instances, robots.txt file looks like this:

# robots.txt for Umbraco
User-agent: *
Disallow: /aspnet_client/
Disallow: /bin/
Disallow: /config/
Disallow: /data/
Disallow: /install/
Disallow: /macroScripts/
Disallow: /masterpages/
Disallow: /umbraco/
Disallow: /umbraco_client/
Disallow: /usercontrols/
Disallow: /xslt/
User-agent: Libwww-perl
Disallow: /
Sitemap: http://{HTTP_HOST}/sitemap

Robots.txt for umbraco

2 Answers2