-1

I have hosting setup on hostgator which creates addon domains inside a subdirectory of the main site. So the structure is

public_html
    - main_site
        - main_site_folders
        - addon_site1
        - addon_site2

and so on. If I disallow addon_site1 and addon_site2 from robots.txt in main site will it stop search engines from indexing the addon sites? I am not able to find a direct answer to this, if somebody can point to me a document that explains this concept without ambiguity, it'll be appreciated.

mehulved
  • 371
  • 2
  • 14
  • May have to change my answer below - do you access your website by http://hostname/main_site ? or are you using a host header to partition main_site against another host? – nevelis Jan 26 '13 at 17:08
  • main_site = main_site.com. main_site/addon_site1 = addon_site1.com main_site/addon_site2 = addon_site.com These are very much independent websites, I'd host each of them in a different folder if that was possible but this is how it has been setup. – mehulved Jan 26 '13 at 17:13

2 Answers2

2

If you block /main_site/addon_site, then robots would not go to www.main_site.com/addon_site. However, that wouldn't prevent them from going to www.addon_site.com directly.

If you want to prevent them from indexing www.addon_site.com, you'll need to put a robots.txt file in the addon_site folder:

User-agent: *
Disallow: /

You need to clarify in your question what you're trying to do.

Jim Mischel
  • 131,090
  • 20
  • 188
  • 351
1

Yes:

Disallow: /main_site/addon_site1
Disallow: /main_site/addon_site2

...assuming robots.txt is in your public_html directory.

If your site root is in /public_html/main_site (eg: http://example.com/ points to /public_html/main_site), you'd put robots.txt in /public_html/main_site with the following:

Disallow: /addon_site1
Disallow: /addon_site2
nevelis
  • 736
  • 6
  • 17