0
  • Front-end Web Server: IIS 7.5
  • Back-end Web Server: Tomcat 5.5.28.
  • Both are on the same machine.
  • IIS uses an AJP Connector -- isapi filter -- to pass appropriate requests to Tomcat.

IIS is using a robots.txt file to prevent web crawlers from accessing web files. The location of it is inetpub\wwwroot\robots.txt. Here is its contents:

 User-agent: *
 Disallow: /

The Tomcat application is located in the inetpub\wwwroot\tomcatapp\ directory. In other words, any requests that contain the tomcatapp directory are forwarded from IIS to Tomcat.

Do I need an additional robots.txt file in the Tomcat context (i.e. tomcatapp) and, if so, where do I put it? I ask because i don't know if the search bots read just the front-end web server or they are capable of reading the back-end web server, too.

Thank you very much for any help.

user717236
  • 4,959
  • 19
  • 66
  • 102

1 Answers1

0

The robots.txt only makes sense in the root folder of your domain. Its not a security aspect at all, only a hint for "friendly" search engines, to let them know wich pages you would like them to index.

Don't place your Tomcats apps inside the wwwroot folder, because if the AJP mapping fails all directories and files will be accessible. Place it elsewhere and map to that directory, so only Tomcat is able to access it.

Stefan
  • 12,108
  • 5
  • 47
  • 66