0

Can I for example, include

User-agent: * Disallow: /

in the robots and then include all of the links i want including the home page in the sitemap and multiple other sitemaps and have them be indexed? Does that work?

I have noticed one of our competitors uses robots to disallow links, and also uses rel=nofollow on those exact same links, but uses extensive sitemaps to make up for it and they are indexed. I have noticed that this method is better for certain sites which are link heavy and require specific order.

Can I do this?

user3283015
  • 435
  • 1
  • 4
  • 7

1 Answers1

0

Pages restricted by robots.txt won't be indexed at all. According to Google Webmaster Tools:

If your site has content you don't want Google or other search engines to access, use a robots.txt file to specify how search engines should crawl your site's content.

Including such URLs to sitemap.xml won't affect on indexing also.

Oleg
  • 823
  • 6
  • 5