Use of Robot.txt

Question

Why will I want a Robot.txt file on my website. I know how to use it, I want to know why it is used?

You can find a good explanation at http://www.robotstxt.org/robotstxt.html — pydvlpr, Feb 10 '18 at 07:19

score -1 · Accepted Answer · answered Feb 10 '18 at 07:21

Surely you can do a bit more research on Google to find the answer but as an example:

Imagine Google finds your new site published online and scans all your pages to publish in its search index. You may not want google to publish every page such as a sub page because you don’t want users to access it directly from google but rather through a main entry page.

Use of Robot.txt

1 Answers1