2

I am currently using robots.txt file that disallows indexing my WP Admin. Currently my robots file looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

I would like the search engines to also STOP indexing my images. These are my private portfolio images and I don't want them to appear in any online search.

So I will be adding the restriction for Google Image bot and Bing bot. My Robots.txt file will now look like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: Googlebot-Image
Disallow: /

User-agent: msnbot-media
Disallow: /

Is this written correctly? Any other suggestions? Are there any other image bots I should be including?

Joe Bloggs
  • 1,410
  • 2
  • 24
  • 53

1 Answers1

2

Your current layout is correct, but there is an all-inclusive way of block robots from crawling your images.

Seeing that you are using WordPress, I am going to assume that you are using the normal file upload structure, which means that you could also add the following to your robots.txt:

User-agent: *
Disallow: /wp-content/uploads/

I would suggest you use this in conjunction with what you currently have as it will only block images that are located in your uploads folder. This will not work for images directly in your theme file, or plugin content etc.

Your full list would look like:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-content/uploads/
Allow: /wp-admin/admin-ajax.php

User-agent: Googlebot-Image
Disallow: /

User-agent: msnbot-media
Disallow: /

This route may seem primitive at first, but keep in mind that there are hundreds of crawlers out there. Creating a rule for each of them would probably cause rather severe headaches.

By blocking the location of where 99.9% of your images are located, you elimate the need to list every crawler that you can find, and you also ensure that any new crawlers are immediately added to the list if they adhere to your robots.txt file.

At this point, it is also worth mentioning that there are quite a few web crawlers that completely ignore your robots file.

Frits
  • 7,341
  • 10
  • 42
  • 60
  • You're welcome @JoeBloggs - glad I could help :) please don't forget to mark the answer as accepted/upvote :) – Frits Jul 13 '17 at 06:35