Can I use robots.txt to send the robots to a specific folder?

Question

So I have a regular website and a blog in the same domain. In the future I plan on buying a domain exclusively for the blog but for now this is the way I'll do it.

The blog is in the directory /blog and there are no links from the main site to the blog. So I presume robots just can't get there.

Is there a way to send robots to a particular directory using robots.txt? Is there any other solution than putting a link in the main site?

Probably more suitable for http://webmasters.stackexchange.com/ — Luc M, Jun 14 '13 at 12:57
Thank you, there are so many stack exchange websites these days... — Pier, Jun 14 '13 at 16:53

score 1 · Accepted Answer · answered Jun 14 '13 at 02:15

You probably want to use sitemaps to achieve that

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.

http://en.wikipedia.org/wiki/Robots_exclusion_standard

Can I use robots.txt to send the robots to a specific folder?

1 Answers1