Robots file behaviour

Question

I've noticed something odd today.

if i go to http://www.google.com/robots.txt, IE11 shows me the contents of google's robot file. However if i go to my site (still in development) using the same browser and point it to robots.txt, IE asks if i want to open or save it to disk, and if i click open, it opens in notepad and not in the browser like google's.

Is there any difference in doing things like google does (obviously there's a difference in the way the file is served, so what are the pro's and con's of either method)? And how do hey manage to do it? I assume AddType/AddHandler have something to do with it but i can't figure out how to do it.

Here's what fiddler gives me for my site's robots.txt file:

HTTP/1.1 200 OK
Date: Thu, 02 Apr 2015 17:16:06 GMT
Server: Apache/2.4.9 (Win64) PHP/5.5.12
X-Powered-By: PHP/5.5.12
Content-Disposition: attachment; filename="robots.txt"
Content-Length: 33
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/plain

User-agent: *
Disallow: /images/

It looks like your PHP application is serving robots.txt, rather than your web server. Check your code. — Michael Hampton, Apr 02 '15 at 17:22

AD7six · Accepted Answer · 2015-04-21T12:42:28.713

On "my site" the file is offered for download because of the content disposition header:

Content-Disposition: attachment; filename="robots.txt"

To not download the file but instead render it in the browser - all that's required is to remove that header.

It's of no real consequence whether the file is served with or without that header, as the intended audience (web-cralwers / bots / Not Humans using a browser) almost certainly ignore it.

Robots file behaviour

1 Answers1