Configuring Django Cache Machine on Heroku Memcachier?

Question

I would like to use the awesome Cache Machine Django app (https://github.com/jbalogh/django-cache-machine) using Memcachier on Heroku.

From what I understand, Cache Machine does not work out of the box with Memcachier because Memcachier requires PyLibMC and SASL authentication (see https://devcenter.heroku.com/articles/memcachier#django). Cache Machine says that it supports PyLibMC -- and that all you have to do is drop in "caching.backends.memcached.PyLibMCCache" to the CACHES setting.

When you do that, though, you get this error: "error 47 from memcached_set: SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY"

I thought the cause of this is that caching.backends.memcached.PyLibMCCache inherits from django.core.cache.backends.memcached.PyLibMC (see https://github.com/jbalogh/django-cache-machine/blob/master/caching/backends/memcached.py), as where it should inherit from django_pylibmc.memcached.PyLibMCCache in order to work on Heroku (but this is sort of a shot in the dark).

I made my own custom cache backend that instead inherited from django_pylibmc.memcached.PyLibMCCache, but now when I check my Heroku Memcachier panel, it doesn't appear that anything I do is increasing the cache -- it's stuck at 50 mb, even though I would expect it to be increasing with each queryset.

Has anyone successfully set up Cache Machine on Heroku? If so, how'd you do it?

Answer 1

I work with MemCachier. We haven't dealt directly with Cache Machine before but I just built a small app then to confirm that it works fine on Heroku and with MemCachier.

Quick background in case you are not very familiar with memcache and MemCachier. There are two protocols for talking between the client and server. One is the older ascii protocol and the other is the newer binary protocol. We at MemCachier only support the binary protocol as it supports authentication while the ascii protocol doesn't.

The mistake you are making is using caching.backends.memcached.PyLibMCCache as the cache backend. While pylibmc is the memcache client we recommend as it supports the binary protocol and sasl authentication, the cache interface that comes with Django sadly doesn't support enabling the binary protocol. So pylibmc simply goes with its default, which is the ascii protocol and fails.

You can see the ticket for Django about this issue here.

As such, at MemCachier we've always recommended that you use the alternative django-pylibmc-sasl package. This package is uses pylibmc as well, but provides a different cache interface than the one django provides that does support enabling the binary protocol and authentication. This is discussed in our documentation here.

Here is the code that I generally place in settings.py to configure Django's cache with MemCachier:'

## MemCachier Settings
## ===================
def get_cache():
  # We do this complicated cache defenition so that on a local machine (where
  # MEMCACHIER_SERVERS won't be defined), the try fails and so we use the
  # inbuilt local memory cache of django.
  try:
    os.environ['MEMCACHE_SERVERS'] = os.environ['MEMCACHIER_SERVERS'].replace(',', ';')
    os.environ['MEMCACHE_USERNAME'] = os.environ['MEMCACHIER_USERNAME']
    os.environ['MEMCACHE_PASSWORD'] = os.environ['MEMCACHIER_PASSWORD']
    return {
      'default': {
        'BACKEND': 'django_pylibmc.memcached.PyLibMCCache',
        'TIMEOUT': None,
        'BINARY': True,
        'OPTIONS': {
          'tcp_nodelay': True,
          'no_block': True,
          'tcp_keepalive': True,
          '_poll_timeout': 2000,
          'ketama': True,
          'connect_timeout': 2000,
          'remove_failed': 4,
          'retry_timeout': 2,
          'dead_timeout': 10
        }
      }
    }
  except:
    # Use django local development cache (for local development).
    return {
      'default': {
        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'
      }
    }

CACHES = get_cache()

This will also make developing on your local machine easy as when MEMCACHIER_SERVERS isn't defined it'll fall back to django's simple local heap cache, avoiding the need for you to install memcache locally.

hope this helps Jake!