django-redis persisting json data

Question

I have a small django site which controls an anstronomy dome and house automation. On start up the project loads 3 json files: relays, conditions and homeautomation. To avoid constant reading and writing to the Pi4's ssd I load the json files into REDIS (on start up in apps, see below). I already have REDIS running in a docker as the project uses celery.

My problem is that within a few minutes of loading the json into REDIS it clears the data out of cache.

I load the json file in the form of a dictionary (dict) in apps

cache.set("REDIS_ashtreeautomation_dict", dict, timeout=None)

and set

CACHES = {
"default": {
    "BACKEND": "django_redis.cache.RedisCache",
    "LOCATION": "redis://redis:6379",
    "OPTIONS": {
        "CLIENT_CLASS": "django_redis.client.DefaultClient",
        "SERIALIZER": "django_redis.serializers.json.JSONSerializer",
        "TIMEOUT": None
    }
}

}

I don't need the data to persist if the dockers go down and I don't need db functions. Caching these files is ideal but I need them to 'stay alive' for the lifetime of the server.

Thank you.

A typical way to handle data with those requirements be to simply load the data into a Python dictionary one time at startup of the process. Is there a reason you need to put it in a "cache", and use Redis? — Kevin Christopher Henry, Jun 23 '22 at 15:16
Thank you Kevin. That is how I originally set up the project, the problem is that a 3rd party app provides sensor data in a json file every 15 seconds. I need to merge that with data from another sensor (every 15 seconds). Plus the data is used across a number of apps so sharing / instantiation is an issue. I already have REDIS running in a docker so this seemed an ideal solution. Particularly when the Django documentation says “TIMEOUT.....TIMEOUT to None so that, by default, cache keys never expire.” I would like to get it working if I can. — SgtBilko, Jun 23 '22 at 15:42
Your problem may simply be that you've put `TIMEOUT` under `OPTIONS`, when it should be at the same level in the hierarchy. See the example configuration in [the documentation](https://docs.djangoproject.com/en/dev/topics/cache/#cache-arguments). Note that `None` isn't a guarantee that your data won't ever get purged. Redis, for example, has its own set of (configurable) eviction policies that it will turn to when it starts to run out of memory. — Kevin Christopher Henry, Jun 23 '22 at 16:01

score 0 · Answer 1 · answered Jun 23 '22 at 17:16

Thank you Kevin.

Moving TIMEOUT solved the issue.

CACHES = {
"default": {
    "BACKEND": "django_redis.cache.RedisCache",
    "LOCATION": "redis://redis:6379",
    "TIMEOUT": None,
    "OPTIONS": {
        "CLIENT_CLASS": "django_redis.client.DefaultClient",
        "SERIALIZER": "django_redis.serializers.json.JSONSerializer",
    }
}

}

I am going to include some code to catch the long term REDIS 'eviction' policies (i.e. reload the json data). I don't want to delve into the REDIS docker.

Thanks

Ian

django-redis persisting json data

1 Answers1