3

I'm using flask-cache in an app and trying to prepopulate the cache in a separate process. The problem is I can't figure out the format the cached values are stored in.

Looking at the cached values they look like they've been pickled, but the values created by cached functions are slightly different from normal pickled values and can't be unpickled directly. Here's an example:

Here is my flask view:

@app.route('/index')
@cache.cached(timeout=60)
def index():
    return 'foo'

Here is the cached value from my view, stored in redis:

>>> r = redis.StrictRedis()
>>> r.keys()
[b'flask_cache_view//index']
>>> r.get('flask_cache_view//index')
b'!\x80\x03X\x03\x00\x00\x00fooq\x00.'

Notice the cached bytestring has a leading '!'. Compare to manually pickling 'foo':

>>> import pickle
>>> pickle.dumps('foo')
b'\x80\x03X\x03\x00\x00\x00fooq\x00.'

The latter can be unpickled, but attempting to unpickle the flask-cache value results in an error "_pickle.UnpicklingError: invalid load key, '!'."

Because I don't fully understand the problem I'm not comfortable implementing a solution (e.g. removing / prepending '!' on all bytestrings). Am I on the right track here?

daveruinseverything
  • 4,775
  • 28
  • 40

1 Answers1

3

Original answer (pre flask 1.0)

According to the werkzeug.contrib.cache.RedisCache code

def dump_object(self, value):
    """Dumps an object into a string for redis.  By default it serializes
    integers as regular string and pickle dumps everything else.
    """
    t = type(value)
    if t in integer_types:
        return str(value).encode('ascii')
    return b'!' + pickle.dumps(value)

def load_object(self, value):
    """The reversal of :meth:`dump_object`.  This might be called with
    None.
    """
    if value is None:
        return None
    if value.startswith(b'!'):
        try:
            return pickle.loads(value[1:])
        except pickle.PickleError:
            return None
    try:
        return int(value)
    except ValueError:
        # before 0.8 we did not have serialization.  Still support that.
        return value

! is used to differentiate integer and other type of values.

Update: Flask 1+ (cachelib.redis.RedisCache)

RedisCache backend is here now but the serialization is moved into a separate class RedisSerializer.

Storing is a little bit different. Comment remained the same but it just pickles everything now.

def dumps(self, value: _t.Any, protocol: int = pickle.HIGHEST_PROTOCOL) -> bytes:
    """Dumps an object into a string for redis. By default it serializes
    integers as regular string and pickle dumps everything else.
    """
    return b"!" + pickle.dumps(value, protocol)
twil
  • 6,032
  • 1
  • 30
  • 28