7

Something that I just thought about:

Say I'm writing view code for my Django site, and I make a mistake and create an infinite loop.

Whenever someone would try to access the view, the worker assigned to the request (be it a Gevent worker or a Python thread) would stay in a loop indefinitely.

If I understand correctly, the server would send a timeout error to the client after 30 seconds. But what will happen with the Python worker? Will it keep on working indefinitely? That sounds dangerous!

Imagine I've got a server in which I've allocated 10 workers. I let it run and at some point, a client tries to access the view with the infinite loop. A worker will be assigned to it, and will be effectively dead until the next server restart. The dangerous thing is that at first I wouldn't notice it, because the site would just be imperceptibly slower, having 9 workers instead of 10. But then it might happen again and again throughout a long span of time, maybe months. The site would just get progressively slower, until eventually it would be really slow with just one worker.

A server restart would solve the problem, but I'd hate to have my site's functionality depend on server restarts.

Is this a real problem that happens? Is there a way to avoid it?

Update: I'd also really appreciate a way to take a stacktrace of the thread/worker that's stuck in an infinite loop, so I could have that emailed to me so I'll be aware of the problem. (I don't know how to do this because there is no exception being raised.)

Update to people saying things to the effect of "Avoid writing code that has infinite loops": In case it wasn't obvious, I do not spend my free time intentionally putting infinite loops into my code. When these things happen, they are mistakes, and mistakes can be minimized but never completely avoided. I want to know that even when I make a mistake, there'll be a safety net that will notify me and allow me to fix the problem.

Ram Rachum
  • 84,019
  • 84
  • 236
  • 374

3 Answers3

5

It is a real problem. In case of gevent, due to context switching, it can even immediately stop your website from responding.

Everything depends on your environment. For example, when running django in production through uwsgi you can set harakiri - that is time in seconds, after which thread handling the request will be killed if it didn't finish handling the response. It is strongly recommended to set such a value in order to deal with some faulty requests or bad code. Such event is reported in uwsgi log. I believe other solutions for running Django in production have similar options.

Otherwise, due to network architecture, client disconnection will not stop the infinite loop, and by default there will be no response at all - just infinite loading. Various timeout options (one of which harakiri is) may end up showing connection timeout - for example, php has (as far as i remember) default timeout of 30 seconds and it will return 504 gateway timeout. Socket disconnection timeout depends on http server settings and it will not stop application thread, it will only close client socket.

If not using gevent (or any other green threads), infinite loop will tend to take up 100% of available CPU power (limited to one core), possibly eating up more and more memory, so your website will work pretty slow and/or timeout really quick. Django itself is not aware of request time, so - as mentioned before - your production environment stack is the way to prevent this from happening. In case of uwsgi, http://uwsgi-docs.readthedocs.org/en/latest/Options.html#harakiri-verbose is the way to go.

Harakiri does print stack trace of the killed proces: (https://uwsgi-docs.readthedocs.org/en/latest/Tracebacker.html?highlight=harakiri) straight to uwsgi log, and due to alarm system you can get notified through e-mail (http://uwsgi-docs.readthedocs.org/en/latest/AlarmSubsystem.html)

Krzysztof Bujniewicz
  • 2,407
  • 21
  • 16
  • The Harakiri option is a step in the right direction, because it stops the server from being stuck, but it doesn't help you find the root of the problem and fix it. What I'd want is to have a stacktrace of the offending worker emailed to me so I could inspect it and fix the problem in the code. – Ram Rachum Apr 29 '13 at 15:00
  • Harakiri does print stack trace and request information, and nginx alarm system allows e-mail notification. Updated answer with links. – Krzysztof Bujniewicz Apr 29 '13 at 15:19
2

I just tested this on Django's development server.

Results:

  • Does not give a timeout after 30 seconds. (this might because its not a production server though)
  • Stays in loading until i close the page.

I guess one way to avoid it, without actually just avoiding a code like that, would be to use threading to have control of timeouts and be able to stop the thread.

Maybe something like:

import threading
from django.http import HttpResponse

class MyThread(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)
    def run(self):
        print "your possible infinite loop code here"

def possible_loop_view(request):
    thread = MyThread()
    thread.start()
    return HttpResponse("html response")
ronniemagatti
  • 1,667
  • 1
  • 10
  • 5
  • Actually, now that i think of, you may want to call the thread = MyThread() in another function so you can actually access it and stop it later.. but still a possible solution? – ronniemagatti Apr 27 '13 at 13:35
  • I really don't understand how your answer solves anything. For one thing, the code needs to finish *before* the response is returned. Secondly, you didn't even show how the thread you created will be automatically stopped. – Ram Rachum Apr 27 '13 at 15:34
  • Oh, you're right, i guess i didn't think this through so much. I'm sorry. – ronniemagatti Apr 28 '13 at 18:24
0

Yes, your analysis is correct. The worker thread/process will keep running. Moreover, if there is no wait/sleep in the loop, it will hog the CPU. Other threads/process will get very little cpu, resulting your entire site on slow response.

Also, I don't think server will send any timeout error to client explicitly. If the TCP timeout is set, TCP connection will be closed.

Client may also have some timeout setting to get response, which may come into picture.

Avoiding such code is best way to avoid such code. You can also have some monitoring tool on server to look for CPU/memory usage and notify for abnormal activity so that you can take action.

Rohan
  • 52,392
  • 12
  • 90
  • 87