Django Cache: Use Memcached and fallback to FileSystem

Question

Simple question:

My server has 1G RAM and 10GB disk-space

I'm using per-site cache, and I want to use as much as possible Memcached, but when its out of space that the cache will be saved in hard disk.

(All site's pages together are about 2GB)

Is there a simple configuration to achive this?

Is it a smart thing to do at all?

Thanks

One way (not sure if the simplest) is to write [custom backend](https://docs.djangoproject.com/en/dev/topics/cache/#using-a-custom-cache-backend) and switch to file system cache if memcached cannot store things any more. Take a look at django [built-in backends](https://github.com/django/django/tree/master/django/core/cache/backends). — alecxe, Mar 19 '13 at 08:04
@AlexanderAfanasiev I know this is the direction, my question is about best practice to implement it — YardenST, Mar 19 '13 at 09:27

krimkus · Accepted Answer · 2014-01-17T22:35:15.820

It sounds like you want Memcached to behave like normal RAM and page to disk when it is full. It doesn't by default, but you can imitate it by writing your own cache backend as @AlexanderAfanasiev mentioned. The implementation would be something along these lines:

First, define three caches:

CACHES = {
    'default': {
        'BACKEND': 'myapp.cache.DoubleCache',
    },
    'memcached': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11211',
    },
    'filecache': {
        'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
        'LOCATION': '/foo/bar',
    }
}

Then, in myapp/cache.py:

from django.core.cache.backends.base import BaseCache
from django.core.cache import get_cache, cache
mem_cache = get_cache('memcached')
file_cache = get_cache('filecache')

class DoubleCache(BaseCache):

    def get(self, key, default=None, version=None):
        result = mem_cache.get(key, default=default, version=version)
        if result:
            return result
        else:
            return file_cache.get(key, default=default, version=version)

    def set(self, key, value, timeout=None, version=None, client=None, _add_only=False):
        memcache_result = mem_cache.set(key, value, timeout=timeout, version=version, client=client, _add_only=_add_only)
        file_result = file_cache.set(key, value, timeout=timeout, version=version, client=client, _add_only=_add_only)
        return memcache_result

This will always store values in both caches. It will retrieve values from Memcached, and if it misses, will try the file_cache. That means that Memcached can manage its own roll-off and only the oldest hits will have to fall back to file_cache. This is what you'd want.

Of course, you'll also have to implement the rest of the cache functions like delete(), get_many(), etc. Hopefully this gets you on the right path.

Tnx for the detailed answer, however won't the set() method will always succeed to store the value in memcahed? it will just drop an older value and store the new one, causing the get method to never reach the file_cache. Correct me if I'm wrong. — YardenST, Mar 20 '13 at 07:24
Yes, set() will store in both. But if memcached fills up (which will happen with 1GB of ram and a 2GB dataset), then half the values will be in both and half will only be in the file_cache. As memcached drops the least used if it fills up, the half in the file_cache will be the least used pages. That's what you want. If you then try to do a get on a page that is rarely used, it will miss on the memcached get and then return the file_cache one. — krimkus, Mar 20 '13 at 15:45
I get it now, I'll try it later to see if it works as expected — YardenST, Mar 20 '13 at 15:56
@krimkus maybe you could also refresh the Memcached cache upon FileSystem hit, that way hopefully you'll have more relevant information in RAM. — Evgeny, Aug 11 '13 at 01:24
Anyone know if this sort of backend has been publicly published anywhere since Django 1.7 and the original answer was posted??? — Fydo, Mar 30 '15 at 14:08

score -1 · Answer 2 · answered Dec 04 '14 at 16:00

Exploit TCP/IP. With minor effort this logic can be extended to create nice multicache backend with fallbacks.

import socket
socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

try:
    socket.connect(('127.0.0.1', 11211))
    socket.close()

    CACHES = {
        'default': {
            'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
            'LOCATION': '127.0.0.1:11211',
        }
    }
except:
    CACHES = { 
        'default': {
            'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
            'LOCATION': '/srv/django_cache',
        }
    }

What if the backend is OK when `settings.py` is loaded but not at request-time? — Andrea Rabbaglietti, Nov 13 '18 at 12:51

Django Cache: Use Memcached and fallback to FileSystem

2 Answers2