0

So I have a VPS (running debian) setup to host a number of sites i'm working on. with the structure like this:

/var/www/www.website1.com
----------------------------------
/var/www/www.website1.com/htdocs
/var/www/www.website1.com/cgi-bin
/var/www/www.website1.com/logs


/var/www/www.website2.com
----------------------------------
/var/www/www.website2.com/htdocs
/var/www/www.website2.com/cgi-bin
/var/www/www.website2.com/logs

etc

My question is where do I place my robots.txt file. Should it be in the htdocs for each site along with the site files. Or can I have a master robots.txt file in one location for the entire VPS. I am trying to stop google from crawling the log folders and any other folders within the website structure that don't need to be accessed by a search engine. At the moment i seem to have gone overkill and google isn't indexing my index.html on one of my sites along with the descriptions within that.

any help would be great.

user2653673
  • 1

2 Answers2

0

You should have a separate robots.txt file for each site, in each htdocs directory. A given robots.txt file only applies to the domain it's loaded from, so each site should have it's own file. In theory, you could set up one shared robots.txt file for all of your sites, but this is generally a bad idea because it makes it impossible to block something on one site without blocking it on all the others.

plasticinsect
  • 1,702
  • 1
  • 13
  • 23
0

Of-course you can have master robots.txt for your vps if your are willing too. If you do that and when you want to block some folder than you can block it with giving entire path of that folder. But suggest you to go with separate robots.txt for every website you have in your hosted vps.

And you asked google not indexing one of your site, for that you just check where you have blocked the bot, if not than add that site to Google Webmaster tool and verify, than Google will definitely index you website. Do same for all websites which is good to handle separate records in all forms for every website even if it is in single vps.

Anyhow for future server requirement just check out with my vps hosting provider who helps in all way to host my websites and application.

Randy
  • 1