0

If I have the following in my .htaccess: (disallow bots from going to /dir1/dir2)

Disallow: /dir1/dir2

And I have in my .htaccess: (when accessing robots.txt, pipe them the data from dir1/dir2/robots.txt)

RewriteCond %{HTTP_HOST} ^.*domain\.com$
RewriteRule ^robots.txt$ dir1/dir2/robots.txt [NC]

Will "domain.com/robots.txt" still be served to the bot, whereas "domain.com/dir1/dir2/robots.txt" remain unreachable to it?

Lakitu
  • 424
  • 1
  • 4
  • 12

1 Answers1

0

Yes, the "domain.com/robots.txt" will still be served to the bot, because the bot doesn't see the organization of your folders in the server, it just sees the page url. So, with this url, it thinks it is in the root folder and as you disabled "/dir1/dir2" and so allowed the root, it can reach the robots.txt file.

Lucas Willems
  • 6,673
  • 4
  • 28
  • 45