0

On Ubuntu 12.04 with mod_wsgi for python 3.

I have a WSGI application (actually just a script) that happens to launch an external program with Popen for each user session (it's actually a small GTK program, I'm experimenting with its HTML5 backend).

There is a Javascript loop on the client side which send a "keep alive" signal each few seconds to the WSGI. If the WSGI did not received any signal for a session for a while, then it will kill the relative process (and delete the session).

This works well, except when I restart/reload Apache or when I edit the WSGI script (which AFAIK automatically reload the app). If I do such a thing, subprocesses are not killed. They still run (they aren't zombies) and all I can do is to kill them manually (the WSGI lost former sessions so it won't kill any "old" process).

So I would like one of the following:

  • A way to notice that the server is stopping/restarting/reloading from the WSGI side, so that it can clean-up it's subprocesses while it knows them
  • Spawned processes should died with mod_wsgi (currently it seems subprocesses are reattached to Init when mod_wsgi is killed/reloaded)

Here is the VirtualHost I use:

<VirtualHost *:80>

WSGIDaemonProcess deckard_qh user=deckard group=deckard threads=5
WSGIScriptAlias / /home/deckard/wsgi/deckard_qh.wsgi
Alias /ressources /home/deckard/ressources

<Directory /home/deckard/wsgi>
    WSGIProcessGroup deckard_qh
    WSGIApplicationGroup %{GLOBAL}
    Order deny,allow
    Allow from all
</Directory>

<Directory /home/deckard/ressources>
    Order deny,allow
    Allow from all
</Directory>

I tried adding WSGIProcessGroup deckard and WSGIApplicationGroup %{GLOBAL} (as per this answer), but it did not change anything. I also added os.setsid() at the start of my WSGI script, but no result.

Community
  • 1
  • 1
Nicolas Delvaux
  • 175
  • 11
  • I feel like it might be more related to how you are launching them in your script. Could you show an example of how your wsgi script is launching processes? – jdi Aug 15 '12 at 18:55
  • Are they definitely running, or are they in defunct (zombie) state? If immediate parent is killed and couldn't wait for process end state and they later finish, then they can hang around as zombies until whole Apache is shutdown. They consume no resources when in that state except for entry in the process table. – Graham Dumpleton Aug 15 '12 at 21:56
  • I reworded the question. I hope it's clearer now. @jdi It's just a simple Popen. If I run the relevant code in a Python script in a shell and that I kill the shell, then the subprocess is also killed. I guess mod_wsgi do some special things with subprocesses. If you still need some code, I will try to shorten what I have and I will post it later. – Nicolas Delvaux Aug 15 '12 at 22:07
  • @GrahamDumpleton Nope, no Zombies here. It seamed these processes are re-attached to init when mod_wsgi die. – Nicolas Delvaux Aug 15 '12 at 22:11
  • Nothing special is done with sub processes by mod_wsgi. The signal handlers that Apache installs and way they are managed can cause issues for sub processes however that inherit the active signal mask. Because of this, the general advice is not to do anything too complicated with sub processes. This actually is good advice all around as far as web applications. – Graham Dumpleton Aug 15 '12 at 22:51

2 Answers2

2

This is just a sketch, but a simple solution (children manage themselves, my favorite kind of child solution!) would be adding a kill timeout to the Popen processes so if they haven't received a 'keep alive' in 5 minutes (or whatever you think is right) they would save state (if appropriate) and terminate?

I guessed gtk_main_quit() would be what you use to terminate the event loop, but in the event I was off base, substitute whatever would terminate your child process =) Also, I'm guessing the gtk even loop might have its own timer function that would be a different implementation than threading, but I wanted to test what I posted.

import datetime
from threading import Timer

# dummy timestamp for testing, gong should be the 
# timestamp of the last keepAlive signal
gong = datetime.datetime(2012, 8, 16, 16, 3, 18, 341121)  

#seconds before kill check is performed
idle = 5

def bringOutYourDead():
    """If the last keep alive time stamp is more than 5 minutes ago, I am Audi 500."""
    stoneDeadIn =  5
    if datetime.datetime.now() - datetime.timedelta(minutes=stoneDeadIn) >= gong:
        # I used print for whatever command for testing
        print('save_state_or_whatever()')
        print('gtk_main_quit()')
    else:
        print("I'm not dead yet!'")
    # recurse this as you see fit

dung = Timer(idle, bringOutYourDead)
dung.start()
Peter Hanley
  • 1,254
  • 1
  • 11
  • 19
  • Indeed, "suicidal" children is a simple workaround. However, this means you have to control the child code in order to add the timeout. So this is not a "universal" solution. I would prefer if these children could die with their parent in the first place ;). But still, if no-one has a better idea, I will probably use something like this. Thanks. – Nicolas Delvaux Aug 17 '12 at 21:44
  • Of course I found it just now, but [this thread](http://stackoverflow.com/a/11318000/602581) claims that if you do this: `p = subprocess.Popen([sys.executable, '/path/to/script.py'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)` the children will die when apache does. – Peter Hanley Aug 18 '12 at 01:57
  • Unfortunately, it does not work here. I suppose this only works with CGI. Nice find though! – Nicolas Delvaux Aug 18 '12 at 08:51
  • I'm using this concept of "suicidal children" since some months now and it works greatly. Thanks! If you are interested, you can find the code in this project: https://launchpad.net/deckard – Nicolas Delvaux May 10 '13 at 09:56
1

Have you tried sudo service apache2 graceful?

The USR1 or graceful signal causes the parent process to advise the children to exit after their current request (or to exit immediately if they're not serving anything). The parent re-reads its configuration files and re-opens its log files. As each child dies off the parent replaces it with a child from the new generation of the configuration, which begins serving new requests immediately.

http://httpd.apache.org/docs/2.4/stopping.html#graceful

Peter Hanley
  • 1,254
  • 1
  • 11
  • 19
  • This might only apply to the processes which apache has spawned, as opposed to the processes which are being spawned from within the wsgi application. Whether the apache children processes are terminated immediately, or "gracefully" shouldn't make a difference in the children of those children. – jdi Aug 15 '12 at 18:56
  • Same as with a _restart_ here. The spawned processes are still running after using graceful. – Nicolas Delvaux Aug 15 '12 at 21:37