data stored by pylibmc cannot be uncompressed in php memcached

Question

We have an application where we are using python to store lots of data in memcached.We are using pylibmc in python and on php side we are using php-memcached library.As a summary

pylibmc v.1.2.3
php-memcached v.2.0.1
libmemcached v1.0.8.

Everything else is fine except when compression comes into play. This is how data is compressed in python

import pylibmc

mem = pylibmc.Client(['10.90.15.104:11211'], binary=True)
mem.set('foo','this is a rather long string. this is a rather '+
'long string. this is a rather long string. this is a rather' + 
'long string. this is a rather long string', 0, 10)

checking in telnet we see some garbled value, which means it was compressed. Now reading it in php.

$memd = new Memcached();
$memd->addServer('10.90.15.104', 11211);
echo $memd->get('foo');

When above is run, we get same garbled value, which means it is not getting uncompressed. pylibmc is using zlib, so accordingly I have changed php's compression type to zlib also. What other setting needs to be done? Please help.

For further reference here are the memcached's output after setting the string in python pylibmc

get foo
VALUE foo 8 40
x+��,V�D��Ē��"����t�⒢̼t=���g\5#
END

And here's memcached's output for string stored using PHP's memcached client:

get foo
VALUE foo 48 44
�x�+��,V�D��Ē��"����t�⒢̼t=���g\5#
END

As you can see there's something fishy in this. Compressed size in pylibmc is 40 bytes and the same data compressed using php-memcached is of 44 bytes. Also notice the flags as 8 when stored using pylibmc and 48 when stored using php-memcached !

Raber · Accepted Answer · 2012-12-20T21:55:03.610

3

i think what you observe is due to the fact that memcache itself does not implement compression so each library does it in their own way, just compare the flags used to indicate compression is in use

as defined by pylibmc

#define PYLIBMC_FLAG_ZLIB (1 << 3) (so this is flag == 8)

and by php-memcached

#define MEMC_VAL_COMPRESSED (1<<4)

#define MEMC_VAL_COMPRESSION_ZLIB (1<<5)

#define MEMC_VAL_COMPRESSION_FASTLZ (1<<6)

so i think unless you are willing to modify one of this libraries to make it's flags consistent with the other then there is no way out

Edit: Ok, so here is the little patch that bring compression support for pylibmc and php-memcached in sync. See my github fork of pylibmc.

Big fat warring - it works only on strings, so if you want to store objects you MUST do de/serialization on your own (JSON).

edited Dec 20 '12 at 21:55

answered Dec 17 '12 at 15:41

Raber

2,040
16
13

wow, thanks for digging that out. but will the modification of those flags do the job? – Shades88 Dec 18 '12 at 06:26
i would say it depends on the type stored as well, for simple things like ints it should work (provided that you will make libraries agree on the type flags and if you change php compression method from FASTLZ to ZLIB), for objects it would be different as i believe php stores them as JSON and python as pickles – Raber Dec 18 '12 at 07:41
well, then (for strings) just unifying compression method and flag should do the trick :) – Raber Dec 18 '12 at 14:29
and you will be my saviour !! :) – Shades88 Dec 18 '12 at 14:44
I tried your solution. But now when I access that item thru php-memcached I get `Warning: Memcached::get(): unknown payload type in uaTestMemcached.php on line 5 bool(false)` – Shades88 Dec 19 '12 at 10:49
1

ok, for whatever reason i thought flags are hex when in fact they are binary, so php seems to set both `MEMC_VAL_COMPRESSED` and `MEMC_VAL_COMPRESSION_ZLIB` (as seen in OP), tried that and got `Memcached::get(): could not decompress value` so this 4 bytes make a difference – Raber Dec 19 '12 at 19:28
sure, now i know what this 4 bytes are - php stores length of string before compression there (not sure why), so pylibmc just needs to mimic that – Raber Dec 20 '12 at 21:08
hey. Your answer didn't help me directly, but set me in correct path. Thanks a lot for that. That library has been forked and author seems to have solved the problem. Here's link if ur interested.https://github.com/gluedig/pylibmc – Shades88 Dec 28 '12 at 13:35
can you please provide a comilation/config strings for all the components? I'm using them on Ubuntu 14 x64 and no problem of that kind. – Alexey Vesnin Jul 30 '15 at 23:14

amolrajoba · Answer 2 · 2012-12-19T08:27:48.643

1

Raber,

As you said I did changes

Changing php compression method from FASTLZ to ZLIB
Changing pyLibmc flag to 1 << 5 in _pylibmcmodule.h file and re-installed pylibmc
```
#define PYLIBMC_FLAG_ZLIB (1 << 5)
```

Are these correct changes? Something more needs to be done? Because Its not working, getting following error

Warning: Memcached::get(): unknown payload type in uaTestMemcached.php on line 5

bool(false)

edited Dec 19 '12 at 08:27

answered Dec 19 '12 at 08:20

amolrajoba

262
1
4
10

sorry, i know zilch about php and it's support for memcache, what is uaTestMemcached.php?, also how do flags look now when you store something compressed via python vs. php? – Raber Dec 19 '12 at 13:07
uaTestMemcached.php is php test script in use here. Here are the memcached's output after setting the string in python pylibmc (after your suggested changes) `get foo` `VALUE foo 8 42` `x+��,V�D��Ē��"��t�⒢̼t=�*�m0<05` `END` – amolrajoba Dec 19 '12 at 15:47
seems that your change in pylibmc did not take effect, i cannot reproduce because php-memcached won't compress anything for me (don't know why) – Raber Dec 19 '12 at 19:06
1

You need to set memcached.compression_threshold to '10' bytes in php.ini, default is 2000bytes `memcached.compression_threshold=10` – amolrajoba Dec 20 '12 at 03:06
@amolrajoba: I know that pain. given 100 bytes in documentation, but it's 2000 by default !! :O – Shades88 Dec 20 '12 at 11:46

data stored by pylibmc cannot be uncompressed in php memcached

2 Answers2