How to store and retrieve a dictionary with redis

Question

# I have the dictionary my_dict
my_dict = {
    'var1' : 5
    'var2' : 9
}
r = redis.StrictRedis()

How would I store my_dict and retrieve it with redis. For example, the following code does not work.

#Code that doesn't work
r.set('this_dict', my_dict)  # to store my_dict in this_dict
r.get('this_dict')  # to retrieve my_dict

Redis is used as a data lake here. If you don't need individual key:value pairs at retrieval time time, you can use Parquet in MinIO / S3, will be faster and more scalable (to petabytes rather than gigabytes). — mirekphd, Nov 21 '22 at 14:52

score 208 · Accepted Answer · edited Jan 29 '17 at 15:44

208

You can do it by hmset (multiple keys can be set using hmset).

hmset("RedisKey", dictionaryToSet)

import redis
conn = redis.Redis('localhost')

user = {"Name":"Pradeep", "Company":"SCTL", "Address":"Mumbai", "Location":"RCP"}

conn.hmset("pythonDict", user)

conn.hgetall("pythonDict")

{'Company': 'SCTL', 'Address': 'Mumbai', 'Location': 'RCP', 'Name': 'Pradeep'}

edited Jan 29 '17 at 15:44

Ihor Pomaranskyy

5,437
34
37

answered Dec 16 '15 at 11:56

PradeepK

2,246
1
12
11

61

if it is nested data structure rather than simply dict, e.g containing some arrays etc. serialzie your data with `json.dumps()` write as string and after retrive from redis user `json.loads()` for deserializing it back to python data structure – andilabs Sep 08 '16 at 10:12
12

`json.dumps()` and `json.loads()` will only work if you are fine with your dictionary keys always being strings. If you aren't then you might consider using pickle. – ryechus Feb 15 '17 at 16:41
8

json is not compatible with bytes so json serilization is not a global solution, e.g., if you have a dict with a bytes value this will not work. – Tommy Feb 01 '18 at 19:28
11

By way of note, the documentation for `hmset` does not tell you this, but it raises a DataError if you try to store an empty dict. – hlongmore Mar 15 '18 at 19:04
2

@Pradeep how we can make the key dynamic. suppose the data is getting inserted every 15 minute so how i can make the key dynamic – ak3191 Oct 09 '18 at 19:45
@ak3191 you can generate a GUID. – lucid_dreamer Nov 11 '18 at 00:53
2

Be warned that the underlying Redis command `HMSET` [has been deprecated](https://redis.io/commands/hmset) (moved to a different function) in version 4.0.0 (July 2017) but `redis-py` still executes it [here in hmset()](https://redis-py.readthedocs.io/en/latest/_modules/redis/client.html#Redis.hmset). – mirekphd Nov 10 '19 at 21:30
2

This method is deprecated. Use hset instead. But with the new function hset. Dictionary cannot be used anymore. – Hung Le Sep 10 '20 at 17:14
this doesn't work with native datetime objects either – glitchwizard Feb 22 '21 at 00:44
pickle can be dangerous if mishandled. Use msgpack for better serialization of data before storing it into redis. – Akhil Mathew Nov 29 '21 at 08:01

score 44 · Answer 2 · answered Sep 01 '15 at 01:39

44

you can pickle your dict and save as string.

import pickle
import redis

r = redis.StrictRedis('localhost')
mydict = {1:2,2:3,3:4}
p_mydict = pickle.dumps(mydict)
r.set('mydict',p_mydict)

read_dict = r.get('mydict')
yourdict = pickle.loads(read_dict)

answered Sep 01 '15 at 01:39

DaveQ

1,642
10
15

19

This is true, but depending on the rate of reads and writes, this may add serious overhead. Pickling is a slow operation – Tommy Feb 01 '18 at 19:21
7

Please note that if this is used with user input your server [is prone to remote code exection](https://davidhamann.de/2020/04/05/exploiting-python-pickle/), `pickle.loads` is should only be used on 100% trusted data – Paradoxis Jul 01 '20 at 09:26
pickle can be dangerous if mishandled. Use msgpack for better serialization of data before storing it into Redis. – Akhil Mathew Nov 29 '21 at 08:02
2

Pickling has also the important down-part that you cannot debug the stored data in Redis as they are binary. – thanos.a Dec 12 '21 at 21:02
In python3 + redis 3.x, i had to use `pickle.dumps(mydict, protocol=0)` and `pickle.loads(str.encode(read_dict))` – Donn Lee Sep 30 '22 at 01:21

score 23 · Answer 3 · answered May 24 '18 at 08:31

As the basic answer has already give by other people, I would like to add some to it.

Following are the commands in REDIS to perform basic operations with HashMap/Dictionary/Mapping type values.

HGET => Returns value for single key passed
HSET => set/updates value for the single key
HMGET => Returns value for single/multiple keys passed
HMSET => set/updates values for the multiple key
HGETALL => Returns all the (key, value) pairs in the mapping.

Following are their respective methods in redis-py library :-

HGET => hget
HSET => hset
HMGET => hmget
HMSET => hmset
HGETALL => hgetall

All of the above setter methods creates the mapping, if it doesn't exists. All of the above getter methods doesn't raise error/exceptions, if mapping/key in mapping doesn't exists.

Example:
=======
In [98]: import redis
In [99]: conn = redis.Redis('localhost')

In [100]: user = {"Name":"Pradeep", "Company":"SCTL", "Address":"Mumbai", "Location":"RCP"}

In [101]: con.hmset("pythonDict", {"Location": "Ahmedabad"})
Out[101]: True

In [102]: con.hgetall("pythonDict")
Out[102]:
{b'Address': b'Mumbai',
 b'Company': b'SCTL',
 b'Last Name': b'Rajpurohit',
 b'Location': b'Ahmedabad',
 b'Name': b'Mangu Singh'}

In [103]: con.hmset("pythonDict", {"Location": "Ahmedabad", "Company": ["A/C Pri
     ...: sm", "ECW", "Musikaar"]})
Out[103]: True

In [104]: con.hgetall("pythonDict")
Out[104]:
{b'Address': b'Mumbai',
 b'Company': b"['A/C Prism', 'ECW', 'Musikaar']",
 b'Last Name': b'Rajpurohit',
 b'Location': b'Ahmedabad',
 b'Name': b'Mangu Singh'}

In [105]: con.hget("pythonDict", "Name")
Out[105]: b'Mangu Singh'

In [106]: con.hmget("pythonDict", "Name", "Location")
Out[106]: [b'Mangu Singh', b'Ahmedabad']

I hope, it makes things more clear.

how you can make the key dynamic – ak3191 Oct 09 '18 at 19:47 — ak3191, Oct 09 '18 at 19:47

score 20 · Answer 4 · edited Nov 20 '22 at 19:58

20

HMSET is deprecated per the Redis docs. You can now use HSET with a dictionary as follows:

import redis

r = redis.Redis('localhost')
    
key = "hashexample" 
entry = { 
    "version":"1.2.3", 
    "tag":"main", 
    "status":"CREATED",  
    "timeout":"30"
}
r.hset(key, mapping=entry)

Caution: very unintuitively, hset won't accept a dictionary (raising an error suggesting it does not accept dictionaries, see [1]) if it is simply passed to the 2nd positional (unnamed) argument. You need to pass the dictionary to a named argument mapping=.

[1] *** redis.exceptions.DataError: Invalid input of type: 'dict'. Convert to a bytes, string, int or float first.

edited Nov 20 '22 at 19:58

mirekphd

4,799
3
38
59

answered Jun 01 '20 at 17:46

Tad Guski

751
7
6

Thanks! I'm trying to find the doc where all this is spelled out. Do you know where it is. For example, what are the two "Nones" for. – NealWalters Sep 25 '20 at 19:34
@NealWalters: See the line on the HMSET command page - https://redis.io/commands/hmset for deprecation warning. – Saransh Singh Oct 30 '20 at 16:53
@Tad Guski, how to you expire hashexample? – Jashwant Mar 30 '23 at 04:25

score 19 · Answer 5 · edited May 01 '22 at 15:47

19

If you want to store a python dict in redis, it is better to store it as json string.

import json
import redis

r = redis.StrictRedis(host='localhost', port=6379, db=0)
mydict = { 'var1' : 5, 'var2' : 9, 'var3': [1, 5, 9] }
rval = json.dumps(mydict)
r.set('key1', rval)

While retrieving de-serialize it using json.loads

data = r.get('key1')
result = json.loads(data)
arr = result['var3']

What about types (eg.bytes) that are not serialized by json functions ?

You can write encoder/decoder functions for types that cannot be serialized by json functions. eg. writing base64/ascii encoder/decoder function for byte array.

edited May 01 '22 at 15:47

Mark Setchell

191,897
31
273
432

answered Jan 31 '18 at 08:09

Saji Xavier

2,132
16
21

I downvoted this because some dicts cannot be serialized to JSON, for example, dicts with a bytes value. – Tommy Feb 01 '18 at 19:27
2

You can write an encoder/decoder function (according to the requirement, eg. base64/ascii encoding) for the types that cannot be encoded/decoded by default. – Saji Xavier Feb 02 '18 at 06:23
@Tommy - even if use hmset/hgetall , you might need to encode/decode types that are not supported by redis. – Saji Xavier Feb 02 '18 at 07:05
1

Disagreeing about " ... latter operation is O(N)." N being the number of fields you have link to the key. Doing N SET/GET or 1 HGET/HSET is the same complexity. See : https://redis.io/commands/hmset Time-wise, HGET/HSET are atomic transaction, and so are performed faster by REDIS. You are just moving the complexity from Redis to Python Code. – ZettaCircl May 17 '19 at 06:02
The advantage of hmset is the possiblity to retrieve only certain sub-parts of the dict. With json we lose that, so this is as good as pickle or other thing. – Jorge Leitao Nov 17 '19 at 19:51
2

The import json is missing. – thanos.a Dec 12 '21 at 21:05
@thanos.a I agree - added it. – Mark Setchell May 01 '22 at 15:48

score 17 · Answer 6 · answered Aug 30 '16 at 06:00

17

Another way: you can use RedisWorks library.

pip install redisworks

>>> from redisworks import Root
>>> root = Root()
>>> root.something = {1:"a", "b": {2: 2}}  # saves it as Hash type in Redis
...
>>> print(root.something)  # loads it from Redis
{'b': {2: 2}, 1: 'a'}
>>> root.something['b'][2]
2

It converts python types to Redis types and vice-versa.

>>> root.sides = [10, [1, 2]]  # saves it as list in Redis.
>>> print(root.sides)  # loads it from Redis
[10, [1, 2]]
>>> type(root.sides[1])
<class 'list'>

Disclaimer: I wrote the library. Here is the code: https://github.com/seperman/redisworks

answered Aug 30 '16 at 06:00

Seperman

4,254
1
28
27

2

By way of note, RedisWorks uses `hmset` under the hood if you set a variable to a dict, and thus if you do `root.something = {}` you will get a DataError, because `hmset` doesn't allow empty dictionaries. I mention this because the documentation for redis doesn't tell you this. – hlongmore Mar 15 '18 at 19:06
Interesting. Yes it does use `hmset`. I will look into this. @hlongmore – Seperman Apr 13 '18 at 21:45
But still, can it support bytes in dictionary ? – ZettaCircl May 17 '19 at 06:04
BTW, Redis.hmset() is deprecated. you should use Redis.hset() instead. – disooqi Feb 21 '21 at 10:53
Yeah I need to fix it when I get a chance... PR's are very welcome too! – Seperman Feb 24 '21 at 02:03

Ohad Lahav · Answer 7 · 2020-05-21T08:43:24.870

8

One might consider using MessagePack which is endorsed by redis.

import msgpack

data = {
    'one': 'one',
    'two': 2,
    'three': [1, 2, 3]
}

await redis.set('my-key', msgpack.packb(data))
val = await redis.get('my-key')
print(msgpack.unpackb(val))

# {'one': 'one', 'two': 2, 'three': [1, 2, 3]}

Using msgpack-python and aioredis

edited May 21 '20 at 08:43

answered May 12 '20 at 17:18

Ohad Lahav

290
3
7

score 6 · Answer 8 · answered Aug 28 '15 at 17:42

The redis SET command stores a string, not arbitrary data. You could try using the redis HSET command to store the dict as a redis hash with something like

for k,v in my_dict.iteritems():
    r.hset('my_dict', k, v)

but the redis datatypes and python datatypes don't quite line up. Python dicts can be arbitrarily nested, but a redis hash is going to require that your value is a string. Another approach you can take is to convert your python data to string and store that in redis, something like

r.set('this_dict', str(my_dict))

and then when you get the string out you will need to parse it to recreate the python object.

he can convert his data to json and store the result in redis — Narcisse Doudieu Siewe, May 29 '16 at 00:01

score 4 · Answer 9 · edited Nov 14 '19 at 12:47

4

An other way you can approach the matter:

import redis
conn = redis.Redis('localhost')

v={'class':'user','grants': 0, 'nome': 'Roberto', 'cognome': 'Brunialti'}

y=str(v)
print(y['nome']) #<=== this return an error as y is actually a string
conn.set('test',y)

z=eval(conn.get('test'))
print(z['nome']) #<=== this really works!

I did not test it for efficiency/speed.

edited Nov 14 '19 at 12:47

Akshay Mulgavkar

1,727
9
22

answered Nov 14 '19 at 11:53

roberto

51
2

score 2 · Answer 10 · answered Jan 23 '20 at 14:55

If you don't know exactly how to organize data in Redis, I did some performance tests, including the results parsing. The dictonary I used (d) had 437.084 keys (md5 format), and the values of this form:

{"path": "G:\tests\2687.3575.json",
 "info": {"f": "foo", "b": "bar"},
 "score": 2.5}

First Test (inserting data into a redis key-value mapping):

conn.hmset('my_dict', d)  # 437.084 keys added in 8.98s

conn.info()['used_memory_human']  # 166.94 Mb

for key in d:
    json.loads(conn.hget('my_dict', key).decode('utf-8').replace("'", '"'))
    #  41.1 s

import ast
for key in d:
    ast.literal_eval(conn.hget('my_dict', key).decode('utf-8'))
    #  1min 3s

conn.delete('my_dict')  # 526 ms

Second Test (inserting data directly into Redis keys):

for key in d:
    conn.hmset(key, d[key])  # 437.084 keys added in 1min 20s

conn.info()['used_memory_human']  # 326.22 Mb

for key in d:
    json.loads(conn.hgetall(key)[b'info'].decode('utf-8').replace("'", '"'))
    #  1min 11s

for key in d:
    conn.delete(key)
    #  37.3s

As you can see, in the second test, only 'info' values have to be parsed, because the hgetall(key) already returns a dict, but not a nested one.

And of course, the best example of using Redis as python's dicts, is the First Test

score 2 · Answer 11 · answered May 28 '21 at 05:33

DeprecationWarning: Redis.hmset() is deprecated. Use Redis.hset() instead.

Since HMSET is deprecated you can use HSET:

import redis

r = redis.Redis(host='localhost', port=6379, decode_responses=True)
r.hset('user:23', mapping={'id': 23, 'name': 'ip'})
r.hgetall('user:23')

score 0 · Answer 12 · answered Jan 16 '19 at 07:25

Try rejson-py which is relatively new since 2017. Look at this introduction.

from rejson import Client, Path

rj = Client(host='localhost', port=6379)

# Set the key `obj` to some object
obj = {
    'answer': 42,
    'arr': [None, True, 3.14],
    'truth': {
        'coord': 'out there'
    }
}
rj.jsonset('obj', Path.rootPath(), obj)

# Get something
print 'Is there anybody... {}?'.format(
    rj.jsonget('obj', Path('.truth.coord'))
)

# Delete something (or perhaps nothing), append something and pop it
rj.jsondel('obj', Path('.arr[0]'))
rj.jsonarrappend('obj', Path('.arr'), 'something')
print '{} popped!'.format(rj.jsonarrpop('obj', Path('.arr')))

# Update something else
rj.jsonset('obj', Path('.answer'), 2.17)

score 0 · Answer 13 · answered Dec 16 '21 at 16:36

In the context of Nameko (Python microservices framework which frequently uses a redis backend) you can use hmset as follows:

import uuid
from nameko.rpc import rpc
from nameko_redis import Redis

class AirportsService:
    name = "trips_service"
    redis = Redis('development')
    @rpc
    def get(self, trip_id):
        trip = self.redis.get(trip_id)
        return trip
    @rpc
    def create(self, airport_from_id, airport_to_id):
        trip_id = uuid.uuid4().hex
        pyDict = {"from":airport_from_id, "to":airport_to_id}
        self.redis.hmset(trip_id, pyDict)
        return trip_id

ariel guzman · Answer 14 · 2023-02-24T18:50:38.537

Lots of good answers but this worked for me.

store dictionary
get dictionary
nested hash instead of mapping the dict as key to field and value to value like other answers above. (see example 1)
get all field/values and go from there as normally you would in a project where you want to dump a dict to a redis hash where the dict is a nested hash. (see example 2)

note: these commands were done in the python repl

if you want

{'field1': 'Hello', 'field2': 'World'}

Use

r = redis.Redis(host="localhost", port=6379, db=0, decode_responses=True)
pdict = {'field1': 'Hello', 'field2': 'World'}
r.hmset("queues_test", pdict)

Also refer to other answers, particularly Saji Xavier's since its simple and works.

If you want a nested hash like

{'queue1': '{"field1": "Hello", "field2": "World"}'

then

# to set
import json
r = redis.Redis(host="localhost", port=6379, db=0, decode_responses=True)
pdict = {'field1': 'Hello', 'field2': 'World'}
pdict_string = json.dumps(pdict)
r.hset("queues_data", "queue1", pdict_string)

# to get a single field value
r.hget("queues_data", "queue1")
# '{"field1": "Hello", "field2": "World"}'

# to get all fields
data =  r.hgetall("queues_data")
# {'queue1': '{"field1": "Hello", "field2": "World"}'
queue1 = data['queue1']
queue1
# '{"field1": "Hello", "field2": "World"}'

result = json.loads(queue1)
result
# {'field1': 'Hello', 'field2': 'World'}
result['field1']
# 'Hello'

Then if you just need the keys/values

list(data.keys())
# ['queue1', 'queue2']

list(data.values())
# ['{"field1": "Hello", "field2": "World"}', '{"field1": "Hello", "field2": "World"}']

Then if you want get the dict back for all values in one line use

lvalues = list(data.values()) 
# ['{"field1": "Hello", "field2": "World"}', '{"field1": "Hello", "field2": "World"}']

[json.loads(x) for x in lvalues]
# [{'field1': 'Hello', 'field2': 'World'}, {'field1': 'Hello', 'field2': 'World'}]

How to store and retrieve a dictionary with redis

14 Answers14

Linked

Related