Best way to store a large python dictionary in a file

Question

I have a python script (script 1) which generate a large python dictionary. This dictionary has to be read by an another script (script 2). Could any one suggest me the best way to write the python dictionary generated by script 1 and to be read by script 2. In past I have used cPickle to write and read such large dictionaries. Is there a beter way to do this?

Another name for a "large dictionary" is a "database". – georg Aug 17 '12 at 06:31 — georg, Aug 17 '12 at 06:31

score 8 · Answer 1 · answered Aug 17 '12 at 05:11

8

shelve will give you access to each item separately, instead of requiring you to serialize and deserialize the entire dictionary each time.

answered Aug 17 '12 at 05:11

Ignacio Vazquez-Abrams

776,304
153
1,341
1,358

1

shelve is great so long as all the keys are strings – John La Rooy Aug 17 '12 at 05:49
@John La Rooy. What if the keys are strings? – st.ph.n Jan 28 '16 at 17:59

score 4 · Answer 2 · edited Dec 17 '17 at 14:22

4

If you want your dictionary to be readable by different types of scripts (i.e. not just Python), JSON is a good option as well.

It's not as fast as shelve, but it's easy to use and quite readable to the human eye.

import json
with open("/tmp/test.json", "w") as out_handle:
    json.dump(my_dict, out_handle)  # save dictionary

with open("/tmp/test.json", "r") as in_handle:
    my_dict = json.load(in_handle)  # load dictionary

edited Dec 17 '17 at 14:22

Stewart Macdonald

2,062
24
27

answered Aug 17 '12 at 05:25

David Cain

16,484
14
65
75

With really large dictionaries, json can run out of memory. – ChaimG Jun 10 '20 at 15:42

Best way to store a large python dictionary in a file

2 Answers2

Linked