1

I've built a website using CherryPy, Apache, and mod_wsgi, and everything's great except for one problem. When a user action calls for an email to be automatically sent, I sometimes use os.fork so that the parent process can return immediately and give the user an "OK" message, while the child process is in charge of sending the email (which may take several seconds, and I don't want the user to have to wait for that).

The problem arises when the child process has finished its job and calls sys.exit(). It seems that mod_wsgi catches the SystemExit exception, dumps the traceback into my Apache error log, and keeps the child process running. I have two problems with this:

  1. It clutters up the Apache error log with a traceback that does NOT represent anything having actually gone WRONG, and

  2. It leaves the process running, wasting system resources. Worse, how many of these sleeping processes are going to accumulate over time?

For the record, the output in the Apache error log looks like this:

mod_wsgi (pid=14900): SystemExit exception raised by WSGI script '/Users/me/myscript.py' ignored.

followed by the traceback leading up to the sys.exit() call.

I guess this explains why the SystemExit exception is caught by mod_wsgi, but that doesn't help me.

It would be great if there were a way to configure mod_wsgi's behaviour in regards to this, yet when I searched for both "SystemExit" and "sys.exit" in the docs, neither one turned up anything.

Searching for "mod_wsgi SystemExit" on this site yields only six posts, and none of them is asking how to allow a child process to exit.

Does anyone know how I can get the child processes to actually exit, and make mod_wsgi NOT dump anything into the Apache error log?

sk1
  • 13
  • 4

1 Answers1

1

Generally speaking, you shouldn't use fork() in code that's hosted in a server process that you didn't create. It often causes weird process lifecycle problems similar to the ones you're describing here. (This goes beyond Apache, or WSGI, or Python; it applies to a wide variety of situations.)

If it's an option, I'd recommend using a separate worker queue to handle asynchronous tasks. For Python, one frequently used option is Celery.

If that's absolutely not an option, you can bypass the SystemExit exception and exit immediately in Python by calling os._exit(). However, be aware this may still cause some odd behavior, and I wouldn't recommend it.

  • Okay, thanks. The absence of prior discussion did make me wonder if I was going about this in an essentially wrong way. Celery looks good, but I wonder if it is more firepower than I need. Would you see anything wrong with just using Python's built-in `threading` module to start a new thread each time I want to send an email? Or might this again lead to the same kind of process lifecycle problems that you mention? – sk1 Dec 29 '15 at 06:43
  • @ssk Best avoided. It's not quite as bad as forking a child process, but creating a thread that outlives the parent request is still pretty weird. –  Dec 29 '15 at 08:43