38

I am running a django application on twisted using the django-on-twisted scripts from this site.

All requests are served by an nginx server which reverse proxies relevant requests to twisted. I have a url setup for an API, which basically just receives get requests and does some processing on the get parameters before sending a response. However, when a specific client is hitting the api, the twisted server just shuts down. Pasted below is the Nginx log:

the.ip.of.client - - [21/Apr/2012:11:30:36 -0400] "GET /api/url/?get=params&more=params HTTP/1.1" 499 0 "-" "Java/1.6.0_24"

The twisted logs show nothing but twisted stops working at this point. By the error code 499, i am assuming that the client closed the connection unexpectedly, which I have no problem with. Whether the client receives the response or not is not important to me. Here is the relevant django view:

def api_url(request):
    if request.GET:
        get_param = request.GET.get('get', [''])[0]
        more_param = request.GET.get('more', [''])[0]
        #some processing here based on the get params
        return HttpResponse('OK')
    else:
        raise Http404

The request from the client is a valid request and does not affect the processing in an adverse way. I have tested it from the shell. When I tried it on the django development server, it crashed in the same way too without leaving any traces of receiving the request. Everything works perfectly well when testing it from the browser. Also, the twisted server works well for all the regular use cases. This is the first time I am facing an issue with it. Any help or pointers will be appreciated.

user1590499
  • 933
  • 3
  • 10
  • 17
tapan
  • 1,776
  • 2
  • 18
  • 31
  • 1
    What does "shuts down" mean? Does it exit cleanly? Does a signal cause it to exit? – Jean-Paul Calderone Apr 21 '12 at 19:32
  • The twisted server does not write anything to the logs. I am pretty sure it is not a clean exit. It just stops working. Any idea of how i might be able to catch the exit signal? – tapan Apr 21 '12 at 20:12
  • 6
    If you're using bash, then `$?` will help. From the bash man page: **? Expands to the exit status of the most recently executed foreground pipeline.** So, for example, `twistd ...; echo $?` – Jean-Paul Calderone Apr 22 '12 at 00:27
  • You can trap and log Python's unhandled exceptions with `sys.excepthook`. See [my answer to another Twisted-related question](http://stackoverflow.com/questions/9674277/python-application-using-twisted-stops-running-after-user-logs-off-of-windows-xp/9675842#9675842) for a code example. – ivan_pozdeev Jul 08 '12 at 17:55
  • I moved to gunicorn and things work fine now. So I really can't test. I'll try to test it over the weekend and let you know if it works. – tapan Jul 09 '12 at 09:11
  • 1
    @tapan just read http://www.peterbe.com/plog/fcgi-vs-gunicorn-vs-uwsgi/ . Maybe you will choose uWSGI :) – b1_ Jul 18 '12 at 08:30
  • Why don't you use the documented solution with "HttpRequest.method" at page https://docs.djangoproject.com/en/dev/ref/request-response/#django.http.HttpRequest.method ? – MUY Belgium Aug 06 '12 at 14:37
  • @MUYBelgium That clearly is not the thing causing the crash. – tapan Aug 07 '12 at 04:52
  • 3
    Here's a question: does it still error out if you actually comment out the "some processing here based on the get params"? If it does, then it is likely that the issue is with Twisted or the server or the configuration. Lacking the relevant wider context, nobody here could diagnose that much further without having hit the same issue themselves in the past. If it does not, then it's related to the code you've omitted in your question so nobody here can help you with the information you've provided in this case either. However, at least that would tell us which extra detail to ask for. – Walter Mundt Aug 15 '12 at 18:15
  • @WalterMundt yes, it does still error if i comment out the processing related code. Basically, whenever django-on-twisted gets a 499 requests, it just dies. – tapan Aug 15 '12 at 22:08
  • Can you turn debug options to max on all parts of the equation: nginx, django, twisted? – f13o Sep 12 '12 at 13:36
  • Not an answer but a alternative suggestion: have you looked at using a Tornado server to serve just the problematic API (Tornado is based in part on Twistd anyway)? I have never managed to crash it yet despite tons of abuse, and putting it behind NGINX is easy, too. http://www.tornadoweb.org/ – jsh Oct 02 '12 at 15:26

2 Answers2

1

There is no 499 http code in rfc. Nginx defines 499 code itself.

When a client sent a request, and closed the connection without waiting for the response, a 499 code occurs. If there're a lot of 499s in your access_log, it's mostly caused by the slow back-ends (too slow for your users to wait). You may have to optimize your website performance.

http://forum.nginx.org/read.php?2,213789,213794#msg-213794

SanityIO
  • 814
  • 6
  • 15
  • The backend is fine actually. The 499 is caused by a bot which is dumping data onto the backend through get variables. It sends the request and immediately closes the connection without waiting for the response. This is acceptable behaviour. I just don't want twisted to crash because of it. I am using gunicorn now and it works perfectly for the same use case. – tapan Sep 03 '12 at 07:30
1
  • you say the problem is from a client hitting a particular url (reproducible?)
  • since it works for you with gunicorn but not django-on-twisted, either the script is not working properly or twisted.web2 is the issue.

please try $ sh init.sh yourdjangoproject stand.

you can also try to modify run.py to catch SystemExit:

import pdb
try:
   # __main__ stuff here.
except (KeyboardInterrupt, SystemExit):
   pdb.set_trace()
dnozay
  • 23,846
  • 6
  • 82
  • 104
  • Thanks for the answer! I am really not in a position to test this out since we have moved to a completely new architecture. But, I'll try and set up an environment for this specific case sometime later and see if your answer helps. – tapan Oct 12 '12 at 16:58