0

Suppose I have a new verison of a website:

 http://www.mywebsite.com

and I have would like to keep the older site in a sub-directory and treat it seperately:

 http://www.mywebsite.com/old/

My new site has a link to the old one on the main page, but not vice-versa.

1) Should I create 2 sitemaps? One for the new and one for the old?

2) When my site gets crawled, how can I limit the path of the crawler? In other words, since the new site has a link to the old one, the crawler will reach the old site. If I do the following in my robots.txt:

 User-agent: *
 Disallow: /old/

I'm worried that it won't crawl the old site (using the 2nd sitemap) since it's blocked. Is that correct?

Maximus
  • 1,441
  • 14
  • 38

1 Answers1

1

1) You could include all URLs in one file, or you could create separate files. One could understand a sitemap as "per (web) site", e.g. see http://www.sitemaps.org/:

In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL

Since you now have two sites, you may create two sitemaps. But again, I don't think that it is strictly defined that way.

2) Well, if you block the URLs in robots.txt, these URLs won't be visited by conforming bots. It doesn't mean that these URLs will never be indexed by search engines, but the pages (= the content) will not.

unor
  • 92,415
  • 26
  • 211
  • 360