Cannot seem able to modify cache value from celery task

Question

Description:

I want to have a cached value (let's call it a flag) to know when a celery task finishes execution. I have a view for the frontend to poll this flag until it turns to False.

Code:

settings.py:

...
MEMCACHED_URL = os.getenv('MEMCACHED_URL', None) # Cache of devel or production
if MEMCACHED_URL:
    CACHES = {
        'default': {
            'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
            'LOCATION': MEMCACHED_URL,
         }
    }
else:
    CACHES = {
        'default': {
            'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
            'LOCATION': 'unique-snowflake',
        }
    }

api/views.py:

def a_view(request):
    # Do some stuff
    cache.add(generated_flag_key, True)
    tasks.my_celery_task.apply_async([argument_1, ..., generated_flag_key])
    # Checking here with cache.get(generated_flag_key), the value is True.
    # Do other stuff.

tasks.py:

@shared_task
def my_celery_task(argument_1, ..., flag_cache_key):
    # Do stuff
    cache.set(flag_cache_key, False) # Checking here with 
                                     # cache.get(flag_cache_key),the
                                     # flag_cache_key value is False

views.py:

def get_cached_value(request, cache_key):
    value = cache_key.get(cache_key) # This remains True until the cache key 
                                     # expires.

Problem:

If I run the task synchronously everything works as expected. When I run the task asynchronously, the cache key stays the same (as expected) and it is correctly passed around through those 3 methods, but the cached value doesn't seem to be updated between the task and the view.

score 6 · Accepted Answer · answered Sep 12 '18 at 12:03

6

If you run your tasks asynchronously, they are part of different processes which means that because of the LocMemCache backend, the task and the view will not use the same storage (each has its own memory).

answered Sep 12 '18 at 12:03

Linovia

19,812
4
47
48

score 2 · Answer 2 · answered Sep 12 '18 at 13:14

Since @Linovia's answer and a dive in Django's documentation, I am now using django-redis as a workaround for my case.

The only thing that needs to change is the CACHES settings (and an active Redis server of course!):

settings.py:

CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": 'redis://127.0.0.1:6379/1',
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
        }
    }
}

Now the cache storage is singular.
django-redis is a well-documented library and one can follow the instructions to make it work.

Cannot seem able to modify cache value from celery task

2 Answers2