How to tell Django, that memcached running with item-size larger than default?

Question

Im using new setting to increase item size in memcached, but i cant store something larger than 1mb through Django backend. I know that memcache module require some setting to achieve thath, and Django use this module in backend.

Answer 1

From Maximum size of object that can be saved in memcached with memcache.py:

There are two entries about that in the memcached FAQ :

• What is the maximum data size you can store? Why are items limited to 1 megabyte in size? The answer to the first one is (quoting, emphasis mine) :

• The maximum size of a value you can store in memcached is 1 megabyte. If your data is larger, consider clientside compression or splitting the value up into multiple keys.

So I'm guessing your 11MB file is quite too big to fit in one memcached entry.

If you do really want to cache larger objects, you will have to subclass Django's MemcachedCache as it doesn't allow you to pass in options:

self._client = self._lib.Client(self._servers, pickleProtocol=pickle.HIGHEST_PROTOCOL)

Example subclass implementation:

from django.core.cache.backends.memcached import MemcachedCache

class LargeMemcachedCache(MemcachedCache):
    "Memcached cache for large objects"

    @property
    def _cache(self):
        if getattr(self, '_client', None) is None:
            self._client = self._lib.Client(self._servers, 
                           pickleProtocol=pickle.HIGHEST_PROTOCOL, 
                           server_max_value_length = 1024*1024*10)
        return self._client

Answer 2

In more recent Django versions you don't need to subclass the cache class, you can instead specify the arguments passed to the cache class's constructor in OPTIONS:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11211',
        'KEY_FUNCTION': 'some.path.to.a.function',
        'OPTIONS': {
            'server_max_value_length': 1024 * 1024 * 10
        }
    }
}

See docs.

Note that you must also increase the size in memcached itself by adding the following line:

-I 10m

to /etc/memcached.conf and restarting it:

sudo service memcached restart