34

Here is my current method of serving robots.txt

url(r'^robots\.txt/$', TemplateView.as_view(template_name='robots.txt',
                                            content_type='text/plain')),

I don't think that this is the best way. I think it would be better if it were just a pure static resource and served statically. But the way my django app is structured is that the static root and all subsequent static files are located in 

http://my.domain.com/static/stuff-here

Any thoughts? I'm amateur at django but 

 TemplateView.as_view(template_name='robots.txt',
                                  content_type='text/plain')

looks a lot more resource consuming than just a static call to my static directory which is served on nginx.

Lucas Ou-Yang
  • 5,505
  • 13
  • 43
  • 62
  • Indeed, it is. Much easier and flexible to deliver robots.txt, or even sitemap.xml via webserver, than coding on Django. Besides, using `TemplateView` for delivering a static file, doesn't seem suitable (it's not a template). And not just coding, but think about all the workflow of code versioning (git....), for something quite easy to serve via webserver. These are my two cents. – ivanleoncz Feb 18 '20 at 15:45

2 Answers2

67

Yes, robots.txt should not be served by Django if the file is static. Try something like this in your Nginx config file:

location  /robots.txt {
    alias  /path/to/static/robots.txt;
}

See here for more info: https://nginx.org/en/docs/http/ngx_http_core_module.html#alias

Same thing applies to the favicon.ico file if you have one.

The equivalent code for Apache config is:

Alias /robots.txt /path/to/static/robots.txt
HankMoody
  • 3,077
  • 1
  • 17
  • 38
  • nice solution. but what happens if I don't have access to that file? – BlaShadow Feb 03 '14 at 15:12
  • @BlaShadow You don't have access to which file? the robot.txt? – KhoPhi May 20 '15 at 15:01
  • 4
    Why should it be served by Nginx? – Daniel Mar 05 '18 at 02:11
  • Easier and flexible to deliver + no need to code anything (besides Pull Requets, git workflow, etc.) on Django urls.py. Just put both files on static folder; run `python manage.py collectstatic`; deliver such files from staticfiles folder on your project via Nginx, Apache, etc. – ivanleoncz Feb 18 '20 at 15:41
13

I know this is a late reply, I was looking for similar solution when don't have access to the web server config. So for anyone else looking for a similar solution, I found this page: http://www.techstricks.com/adding-robots-txt-to-your-django-project/

which suggests adding this to your project url.py: 

from django.conf.urls import url
from django.http import HttpResponse

urlpatterns = [
    #.... your project urls
    url(r'^robots.txt', lambda x: HttpResponse("User-Agent: *\nDisallow:", content_type="text/plain"), name="robots_file"),
]

which I think should be slightly more efficient that using a template file, although it could make your url rules untidy if need multiple 'Disallow:' options.

Stephen Bridgett
  • 621
  • 9
  • 9