pylibmc: 'Assertion "ptr->query_id == query_id +1" failed for function "memcached_get_by_key"'

Question

I have a python web app that uses the pylibmc module to connect to a memcached server. If I test my app with requests once per second or slower, everything works fine. If I send more than one request per second, however, my app crashes and I see the following in my logs:

Assertion "ptr->query_id == query_id +1" failed for function "memcached_get_by_key" likely for "Programmer error, the query_id was not incremented.", at libmemcached/get.cc:107

Assertion "ptr->query_id == query_id +1" failed for function "memcached_get_by_key" likely for "Programmer error, the query_id was not incremented.", at libmemcached/get.cc:89

Any idea what's going wrong or how to fix it?

My code looks like this:

self.mc = pylibmc.Client(
    servers=[os.environ.get(MEMCACHE_SERVER_VAR)],
    username=os.environ.get(MEMCACHE_USER_VAR),
    password=os.environ.get(MEMCACHE_PASS_VAR),
    binary=True
    )

#...

if (self.mc != None):
    self.mc.set(key, stored_data)

#...

page = self.mc.get(key)

Why did you immediately post an answer to your own question? — grc, Sep 14 '12 at 07:33
@grc Because I solved the issue? AFAIK [answering your own question is encouraged](http://blog.stackoverflow.com/2011/07/its-ok-to-ask-and-answer-your-own-questions/). Are you saying I should not? I'm open to other suggestions if other people have them, but my answer solved my issue. When I originally searched for answers to this problem I found none, so I'm hoping that posting it here helps others who run into it. — culix, Sep 14 '12 at 15:01

score 4 · Accepted Answer · answered Sep 14 '12 at 07:32

This is a threading issue. pylibmc clients are not thread-safe. You should convert your code to use a ThreadMappedPool object to ensure you keep a separate connection for each thread. Something like this:

mc = pylibmc.Client(
    servers=[os.environ.get(MEMCACHE_SERVER_VAR)],
    username=os.environ.get(MEMCACHE_USER_VAR),
    password=os.environ.get(MEMCACHE_PASS_VAR),
    binary=True
    )
self.pool = pylibmc.ThreadMappedPool(mc)

#...

if (self.pool != None):
    with self.pool.reserve() as mc:
        mc.set(key, stored_data)

#...

if (self.pool != None):
    with self.pool.reserve() as mc:
        page = mc.get(key)

Make sure to call self.pool.relinquish() when the thread is finished, possibly in the destructor!

(In my case this happened because I was using cherrypy as my web server, and cherrypy spawns 10 separate threads to serve requests by default.)

score 1 · Answer 2 · answered Feb 23 '15 at 22:30

1

I ran into the same issue running Django on Apache. Switching from pylibmc to python-memcached eliminated the problem for me.

answered Feb 23 '15 at 22:30

SMX

1,372
15
14

Also, see the django_elasticache.memcached.ElastiCache backend. It uses pylibmc, but calls it in such as way so it plays nicely with the GIL and doesn't crash. – Cerin Jul 16 '16 at 16:36

pylibmc: 'Assertion "ptr->query_id == query_id +1" failed for function "memcached_get_by_key"'

2 Answers2

Linked