Django global data for threads

Question

I have a shared global data object in my single-process multi-threaded django server - an object which is frequently used, but calculated infrequently. The calculation is time-consuming, so I want to share the results.

I thought it would work to use django's LocalMemCache for this simple data. Oddly, it seems to work for multiple ajax calls on a single page load, but for some reason, when I reload the page in my browser, the cache is empty again.

What am I doing wrong?

Is there a better way? Would a global variable be just as efficient, if I control write-access with a thread lock?

Here's basically what I'm doing:

from threading import Lock
from django.core.cache import get_cache

my_lock = Lock()
cache = get_cache('default')

def get_global_data():
    my_lock.acquire()
    try:
        cached_data = cache.get('data')
        if not cached_data:
            cached_data = big_function_to_calculate_data()
            cache.set('data', cached_data)
    finally:
        my_lock.release()
    return cached_data

# settings.py defines my django LocMemCache as:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
        'LOCATION': 'my_server'
    }
}

Edit:

The root cause of the problem was retrieving data based on an access control list (not part of the code here), which varied based on request type (GET, POST), etc. When calculating, this was a POST request with one set of access, and when reading it was a GET request with a different set of access, and was returning a different (and invalid) set of results.

Probably a silly question, but are you sure your server is running as single-process? Can you post the configuration? — Daniel Roseman, May 12 '14 at 14:52
In production yes, but I'm getting this behavior in my development environment, which is just `manage.py runserver 0.0.0.0:8000` I always assumed this was single process, multiple threads. — Adam Morris, May 12 '14 at 15:08
What about encapsulating the calculation into a model/view, and storing the results in a file that's referenced in that model? — huu, May 12 '14 at 15:09

score 3 · Accepted Answer · answered May 12 '14 at 15:44

3

The above works. As a side note, using a persistent database cache seems to be preferred over LocMemCache.

# run python manage.py createcachetable

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.db.DatabaseCache',
        'LOCATION': 'my_cache_table'
    }
}

The root cause of the problem was retrieving data based on an access control list, which varied based on request type (GET, POST), etc. When calculating, this was a POST request with one set of access, and when reading it was a GET request with a different set of access.

answered May 12 '14 at 15:44

Adam Morris

8,265
12
45
68

Are you sure this works between threads? From my logs I see a cache hit from the same thread but a cache miss from a different thread. – sureshvv Sep 22 '16 at 17:48
I've been using `django.core.cache.backends.locmem.LocMemCache` for some time in a production environment, and it appears to be working between threads. I've never tried a database cache -- as the information is limited and we're on a single server. – Adam Morris Sep 23 '16 at 18:36

Django global data for threads

1 Answers1

Linked