1

I am trying to queue tasks for Python RQ using JavaScript. To do that I just monitored what RQ does with Redis. In particular it stores a pickled list indicating where to find the task to run including input args etc. Here I am using the example found on RQ's homepage. When storing the following string (pickled list):

'\x80\x02(X\x1c\x00\x00\x00my_module.count_words_at_urlq\x01NU\x0fhttp://nvie.comq\x02\x85q\x03}q\x04tq\x05.'

with JavaScript, it does something weird with the literals \x80 and \x85. It replaces them with \xc2\x80 and \xc2\x85 and I am unable to figure out why and how to stop this behavior. The rest of the string gets funneled into Redis as expected. The additional literal hinders the rqworker, more specificly pickle, to function correctly.

> encodeURI('\x80\x02(X\x1c\x00\x00\x00my_module.count_words_at_urlq\x01NU\x0fhttp://nvie.comq\x02\x85q\x03}q\x04tq\x05.')
'%C2%80%02(X%1C%00%00%00my_module.count_words_at_urlq%01NU%0Fhttp://nvie.comq%02%C2%85q%03%7Dq%04tq%05.'

> decodeURI('\x80\x02(X\x1c\x00\x00\x00my_module.count_words_at_urlq\x01NU\x0fhttp://nvie.comq\x02\x85q\x03}q\x04tq\x05.')
' \u0002(X\u001c\u0000\u0000\u0000my_module.count_words_at_urlq\u0001NU\u000fhttp://nvie.comq \u0002q\u0003}q\u0004tq\u0005.'

I am using node and node_redis.

honk
  • 9,137
  • 11
  • 75
  • 83
Da Kr
  • 41
  • 4

1 Answers1

1

Found it out on my own. Python's pickle output string is in fact binary which is the common way to write and read files in Python. Javascript does not like that and "fixes" it. The solution is Javascript's buffer.

> new Buffer('\x80\x85', 'binary')
<Buffer 80 85>

whereas

> new Buffer('\x80\x85')
<Buffer c2 80 c2 85>

Just feed node_redis the buffer object, the "correct" representation of the serialized object will appear in redis and rqworker works as expected. Keep in mind that the 'binary' mode is in the process of being deprecated.

Da Kr
  • 41
  • 4