Why are my mod_python global variables resetting?

Question

I have a server using apache2 with mod_python that seems to reset global variables after some short period of time. Until this happens, all the features of the server and global variables are handled exactly as I would expect.

To rule out the possibility of some runtime error (although I do have debugging on and don't see any errors), I made this simplified script to demonstrate how a global variable is being reset. It just holds a global counter and increments it every time the server is accessed:

from mod_python import apache

counter = 0

def handler(req):
    global counter
    counter += 1

    req.content_type = 'text/plain'
    req.write('counter: '+str(counter))

    return apache.OK

I can keep refreshing the page and watch the counter go up as expected. However, at some point the counter jumps back to 1.

I tried simply holding the F5 key down to rapidly refresh the page and see if it would still reset while I was continually accessing it, and I noticed something else. At first it would still drop down to 1 a few times, but eventually it would drop down to other numbers. For example I got up to around 200 refreshes and it just dropped down to around 100, and would occasionally even jump upward back into the 200s.

It seems like there are multiple virtual shells for the server script being started, and the server just randomly switches between them or starts new ones. How can I prevent this?

abarnert · Accepted Answer · 2014-09-19T23:46:20.740

It seems like there are multiple virtual shells for the server script being started, and the server just randomly switches between them or starts new ones.

Exactly.

Apache has a variety of different things that it does to maximize concurrency, one of which is to spawn a bunch of child processes.

With mod_python, each child process gets its own independent Python interpreter, which means they each have their own copies of any global variables.

How can I prevent this?

Well, you could disable forking in Apache. But this is a bad idea. Unless you really know what you're doing, this will kill your scalability. And it still won't help if, say, you restart the server (which Apache will sometimes do to itself to clear up memory leaks, unless you configure it not to).

Web services are supposed to be stateless—or, rather, if they have any state, they're supposed to package it up and persist it between requests. For per-user state, you can store it in a cookie, or a hidden form field, and get their browser to pass it back. But for system-wide state, that doesn't work; you need to save it to something external.

The usual solution is a database. But for something this simple, you could use anything—even a plain text file and a flock. Which is verbose, but easy to understand without having to learn SQL or some other database interface, and it makes you think about the concurrency issues (and understanding them is critical), so I'll show that:

import contextlib
import fcntl
from mod_python import apache

@contextlib.contextmanager
def flocking(f, flag=fcntl.LOCK_EX):
    fcntl.flock(f, flag)
    try:
        yield f
    finally:
        fcntl.flock(f, fcntl.LOCK_UN)

def bump_counter():
    while True:
        try:
            with flocking(open('storage.lock', 'r+')) as f:
                val = int(f.read())
                f.seek(0)
                f.write(str(val))
                return val
        except OSError:
            pass
        try:
            with flocking(open('storage.lock', 'x')) as f:
                val = 0
                f.write(str(val))
                return val
            except OSError:
                pass

def handler(req):
    req.content_type = 'text/plain'
    req.write('counter: '+str(bump_counter()))

    return apache.OK

Most of the code is about error handling, mainly to deal with the case that this is the first-ever request, so the file doesn't exist yet. In that case, we try to open it in x mode, which will only succeed if the file doesn't exist, just in case two requests come in right as you start up and they both think they're first. This way, one of them will fail, and will go back and try the whole loop again.

In real life, you'll want better error handling, because "file not found" isn't the only reason this could fail, and you don't want the server to just block forever when that happens. But in real life, you're probably going to use a database that takes care of this for you.

Thanks for the reply. I was hoping for the data to be stored in RAM though which would make the server more responsive than having to read and write to files. Is there any way to do that? — spacelike, Sep 19 '14 at 23:46
@spacelike: First, test before you optimize. The file is likely to stay in the kernel's disk cache 99.99% of the time. Also, remember that if you don't use the disk, every time you restart the server you'll be resetting to 0. If you really do need it to be in RAM, it has to be _shared_ RAM. You can use `mmap` (with a non-persistent file, which forces it to stay in cache), or `shm`. Or you can write a helper process that keeps track of the state for everyone else and use IPC from the `mod_python` processes. Or you can use something like `memcached` that wraps it all up for you. — abarnert, Sep 19 '14 at 23:53
@spacelike: And again, the standard solution is a database. Databases like postgresql or MySQL, or various no-SQL alternatives, are really good at caching data, and they take care of locking, persistence, and everything else for you. It might seem like overkill for just storing a single variable, but it's not much more work to do it right, and it gives you the flexibility (and the skill set) to expand later. — abarnert, Sep 19 '14 at 23:55
Thanks for the advice that's extremely helpful. The actual server is more of a forum so it would store a lot more data than just a single variable, the simple counter variable I posted was just an example So using a database sounds like a perfect solution. Thank you — spacelike, Sep 20 '14 at 00:16

score 0 · Answer 2 · answered Sep 19 '14 at 23:30

mod_python runs in the context of the web server. You must create only a single server process to handle all incoming requests, and to make sure that it never dies. Therefore you must set the value of the MaxClients directive to 1 and the value of the MaxRequestsPerChild directive to 0.

MaxClients 1
MaxRequestsPerChild 0

Of course, the previous paragraph is completely facetious since doing so will drive server performance into the ground. The correct solution is to not use global variables with mod_python for any data you cannot afford to have disappear underneath you.

Why are my mod_python global variables resetting?

2 Answers2