3

I am building some very large lookup tables in Redis. I have some straightforward code that works as intended when looping through a dict to set a single value in my Redis hash (via a pipeline) for each item using hset():

foo = {"1234": "5678", "abcd": "efgh", ... }

with self.db.pipeline() as pipe:
    for foo in bar:
        pipe.hset("lookup_table", foo["key"], foo["value"])
pipe.execute()

This is slow with a large dict. To speed it up, I want to be able to set multiple items as a mapping into the pipeline without having to loop over it. With hmset() now deprecated, it seems that hset() can accept a mapping via a keyword arg. I have attempted to do the following:

with self.db.pipeline() as pipe:    
    pipe.hset("lookup_table", mapping=foo)
pipe.execute()

but this yields the error TypeError: hset() got an unexpected keyword argument 'mapping'. Am I using hset() incorrectly? Or am I mistaken in thinking that hset() can accept multiple items in this way?

I'm using py-redis 3.4.1 with Python 3.7.5.

Harry
  • 4,660
  • 7
  • 37
  • 65

3 Answers3

3

This appears to be a known issue as shown here --> https://github.com/andymccurdy/redis-py/issues/1310#issuecomment-603081122.

As you can see in that image linked, the source code in PyPi has hset with a function signature that does not include the keyword mapping. You should verify in your installation of py-redis that the same issue is present and follow that ticket as well. To work around it you can clone straight from master branch in order to use that feature.

gold_cy
  • 13,648
  • 3
  • 23
  • 45
  • Thanks, I had reviewed the source code on Github and saw that it was present in 3.4.1 and that I had 3.4.1 installed via pip, so thought it was the code that I was using. Installing from the Github repo solved the issue (and it is indeed much faster). – Harry Apr 10 '20 at 12:13
2

Updating with

pip install -U redis

solved the issue for me.

stefanbschneider
  • 5,460
  • 8
  • 50
  • 88
0

For anyone coming later to find a better answer and example.

# test mapping to add a full dict
foo = {"1234": "5678", "abcd": "efgh" }
queueLib.redis.hset("queues_data2", mapping=foo)
# works

import json
foo2 = {"1234": "5678", "abcd": {} }
queueLib.redis.hset("queues_data2", mapping=foo2)
# fails with
# redis.exceptions.DataError: Invalid input of type: 'dict'. Convert to a bytes, string, int or float first.

# if you try via serialization
pdict_string = json.dumps(foo2)
queueLib.redis.hset("queues_data2", mapping=pdict_string)
# fails with 
# AttributeError: 'str' object has no attribute 'items'

# now try 
update_nested_dict = json.dumps(foo2['abcd'])
foo2['abcd'] = update_nested_dict
queueLib.redis.hset("queues_data2", mapping=foo2)

# you can also loop through each value of the dict and serialize it with something like:

foo3 = {"1234": {}, "abcd": {} }
{k: json.dumps(v) for k,v in foo3.items()}
# {'1234': '{}', 'abcd': '{}'}

Also covered more on handling dict with redis here

ariel guzman
  • 590
  • 7
  • 11