Django-redis-cache failing to fetch data from redis

Question

I have a question regarding the working of this package. How does it writes to redis db ?

This is my settings for redis -

CACHES = {
     'default': {
    'BACKEND': 'redis_cache.RedisCache',
    'LOCATION': '/var/run/redis/redis.sock',
    'OPTIONS': {
        'DB': 2,
    },
},
}

This is my views file,

def postview(request):
    print("Working")
   #post_list = Post.objects.all()
    if cache.get("posts") == None:
            post_list = Post.objects.all()
            print("going to be cached")
            a = cache.get("aman")
            print("aman ", a)
            cache.set("posts", post_list, timeout=60*100*10)
            print("cached")
    else :
            post_list = cache.get("posts")
            aman = cache.get("aman")
            print(aman, " aman's job")
            print("already present in cache")
    context = {"post_list" : post_list}
    print("Problem")
    return render(request, 'post_list.html', context)

@cache_page(60*15*10, key_prefix="cache_redis")
def testview(request):
    post_list = cache.get("posts")
    print("post_list is", post_list)
    return render(request, 'post_list.html', {"post_list":post_list})

@cache_page(60*25*10, key_prefix="cache_test")
def new(request):
    print("Hey")
    print("cache_page is working")
    return HttpResponse("Hello, I am Mohammed")

This is my redis -cli ,

luvpreet@DHARI-Inspiron-3542:~/test_venv_wrapper/test_redis/cache_redis$ redis-cli
127.0.0.1:6379> select 2
 OK
127.0.0.1:6379[2]> set "a" "aman"
 OK
127.0.0.1:6379[2]> set ":1:a" "theman"
 OK
 127.0.0.1:6379[2]> keys *
 1) "a"
 2)":1:views.decorators.cache.cache_page..GET.ad00468064711919773512f81be0dbc4.d41d8cd98f00b204e9800998ecf8427e.en-us.UTC"
 3) ":1:posts"
 4) ":1:a"
 5) ":1:aman"
 6) ":1:views.decorators.cache.cache_header.cache_test.ad00468064711919773512f81be0dbc4.en-us.UTC"
 7) ":1:views.decorators.cache.cache_header..ad00468064711919773512f81be0dbc4.en-us.UTC"
 8) ":1:b"
 9)":1:views.decorators.cache.cache_page.cache_test.GET.ad00468064711919773512f81be0dbc4.d41d8cd98f00b204e9800998ecf8427e.en-us.UTC"
10) "aman"
127.0.0.1:6379[2]> get ":1:a"
"theman"
127.0.0.1:6379[2]> get "a"
"aman"

This is the corresponding redis-cli monitor

luvpreet@DHARI-Inspiron-3542:~/test_venv_wrapper/test_redis/cache_redis$ redis-cli monitor
OK
1491412249.001149 [0 unix:/var/run/redis/redis.sock] "INFO"
1491412249.086196 [0 127.0.0.1:44984] "select" "2"
1491412250.001249 [0 unix:/var/run/redis/redis.sock] "INFO"
1491412257.001426 [0 unix:/var/run/redis/redis.sock] "INFO"
1491412257.423536 [2 127.0.0.1:44984] "set" "a" "aman"
1491412258.001311 [0 unix:/var/run/redis/redis.sock] "INFO"
1491412269.001211 [0 unix:/var/run/redis/redis.sock] "INFO"
1491412269.820886 [2 127.0.0.1:44984] "set" ":1:a" "theman"
1491412270.000741 [0 unix:/var/run/redis/redis.sock] "INFO"
1491412272.955386 [2 127.0.0.1:44984] "keys" "*"
1491412273.001121 [0 unix:/var/run/redis/redis.sock] "INFO"
1491412340.991928 [2 127.0.0.1:44984] "get" ":1:a"
1491412341.002001 [0 unix:/var/run/redis/redis.sock] "INFO"
1491412344.106985 [2 127.0.0.1:44984] "get" "a"
1491412345.001677 [0 unix:/var/run/redis/redis.sock] "INFO"

This means that data which I am manually inserting into the database 2 is available and I can fetch that through redis-cli.

But when I try to fetch this manually entered data from the django-app python shell, This happens,

>>> from django.core.cache import cache
>>> cache.get("a")
Traceback (most recent call last):
   File "<console>", line 1, in <module>
   File "/usr/local/lib/python2.7/dist-packages/redis_cache/backends/base.py", line 33, in wrapped
     return method(self, client, key, *args, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/redis_cache/backends/base.py", line 259, in get
     value = self.get_value(value)
   File "/usr/local/lib/python2.7/dist-packages/redis_cache/backends/base.py", line 210, in get_value
     value = self.deserialize(value)
   File "/usr/local/lib/python2.7/dist-packages/redis_cache/backends/base.py", line 197, in deserialize
     return self.serializer.deserialize(value)
   File "/usr/local/lib/python2.7/dist-packages/redis_cache/serializers.py", line 42, in deserialize
     return pickle.loads(force_bytes(value))
UnpicklingError: could not find MARK

Here is the redis-cli monitor corresponding to it,

OK
1491413058.004167 [0 unix:/var/run/redis/redis.sock] "INFO"
1491413059.002746 [0 unix:/var/run/redis/redis.sock] "INFO"
1491413060.663292 [2 unix:/var/run/redis/redis.sock] "GET" ":1:a"
1491413061.001167 [0 unix:/var/run/redis/redis.sock] "INFO"

Why cannot I access the data written manually ? I know that it adds a prefix to the data written through django. The prefix is ":1:key_name". That's why I added 2 keys namely "a" and ":1:a". So that when I try to access the "a", it will call ":1:a".

But this error comes up. So, it definitely might be writing data to redis by some other way. Please tell me about the error and also tell me the way in which it writes data.

DhruvPathak · Answer 1 · 2017-04-05T18:36:11.913

The error you are gettting is not a data retreival error, the data is being retreived, but it is not in correct pickle format.

When you are setting data manually, it is being set in byte format, not pickled format.
When you set data through django-redis, your data is first serialized using cPickle, and then stored into redis as pickled string.
When you retreive the data stored through django-redis, it is retreived as the pickle serialized string, and then deserialized into corresponding python datatype.
Your manually entered data is string type, but not in correct pickle format, hence though it gets retreived, it cannot be converted into corresponding python type, and you get the pickling error.
The solution is to use django-redis itself in the django shell or your code to store and retreive data, entering data manually breaks the serialization contract.

Example:

>>> from django.core.cache import cache
>>> cache.set("THE_KEY","aman")
True
>>> cache.get("THE_KEY")
'aman'
>>> 
dhruv@dhruvpathak:~$ redis-cli
127.0.0.1:6379> keys *
 1) ":1:THE_KEY"
127.0.0.1:6379> get ":1:THE_KEY"
"\x80\x02U\x04amanq\x01."
127.0.0.1:6379> set "THE_MANUAL_KEY" "aman"
OK
127.0.0.1:6379> get "THE_MANUAL_KEY" 
"aman"

and that is that !!! well explained !! I put a issue on there github just to know these steps. I didn't know that data was serialized before it is cached(through cache_page decorator). This explains my every doubt. Thanks dhruv bhai — Luv33preet, Apr 05 '17 at 19:55
I did this now, `a="aman"`; `import cPickle`; `a=cPickle.dumps(a)`; this is the value of a now `"S'aman backend'\np1\n."` I copied this a and did this in redis-cli `set "key_name" "value"` . This value was the copied value of a(from python code). And now in shell, I tried to retrieve it. And this error occured. `django.core.exceptions.ImproperlyConfigured: Requested setting CACHES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings. ` I cannot copy full trace as it is lengthy. — Luv33preet, Apr 05 '17 at 20:09

Django-redis-cache failing to fetch data from redis

1 Answers1