A robots.txt file is usually just a text file under your site root directory. For example, you can view www.amazon.com/robots.txt. But today, I found a website with a strange robots.txt file. If you just type
http://xli.bugs3.com/robots.txt
it does not show a text file, instead it still shows the home page of that site. How could it happen and why does the webmaster do this?
Assuming a fairly conventional/basic server setup, where it is just files as you say, it could simply be a htaccess redirect rule. The rule might be something like "serve a file if it's on the server, otherwise just serve the index".
Or it might be an application server like Rails, where there's no direct relationship between the server directory structure and URL pathnames.
