1

I have a large dictionary object dict_tmp that takes 40GB in RAM (system has a total of 64GB), which has string keys and float values. I use d = shelve.open(fname, protocol=2) and d['dict_tmp'] = dict_tmp to save the dictionary, which produces the foll error:

Traceback (most recent call last):
  File "file.py", line 160, in <module>
    d['dict_tmp'] = dict_tmp
  File "/usr/lib/python2.7/shelve.py", line 133, in __setitem__
    self.dict[key] = f.getvalue()
  File "/usr/lib/python2.7/bsddb/__init__.py", line 279, in __setitem__
    _DeadlockWrap(wrapF)  # self.db[key] = value
  File "/usr/lib/python2.7/bsddb/dbutils.py", line 68, in DeadlockWrap
    return function(*_args, **_kwargs)
  File "/usr/lib/python2.7/bsddb/__init__.py", line 278, in wrapF
    self.db[key] = value
TypeError: Data values must be of type string or None.

I believe shelve module is for general python objects but I think this issue may be related to anydbm. Any help would be really appreciated!

Question aside: If not shelve (as answered here and here), which is the best way to save large dictionaries? Thank you so much!

Community
  • 1
  • 1
  • What about the other options mentioned in the linked posts, JSON and SQLite? – agf Jul 27 '13 at 00:20
  • 2
    4 questions. What's in the dict? What's the difference between `d` and `dict`? Why are you calling a variable `dict`? How much memory do you have if you can spend *40 entire gigabytes* on a dict? – user2357112 Jul 27 '13 at 00:24
  • Thank you @agf, I haven't tried the other options. Shelve is helpful for dealing with general python objects. I hope this can be fixed within the shelve module, but otherwise I would've to move. –  Jul 27 '13 at 00:25
  • @user2357112, sorry for using the python keyword `dict` - I just renamed it from the actual code. I've changed the question to reflect your inputs. Thanks! –  Jul 27 '13 at 00:31
  • 1
    I think you might have missed the point of `shelve`. A shelf is supposed to be a persistent replacement for a dict - that is, you store the data directly to the shelf instead of a real dict, and when the shelf is closed, your data is all on disk. If you want to load an entire giant dict from disk to memory and store it back all at once when you're done, you could just pickle and unpickle it with `cPickle`. – user2357112 Jul 27 '13 at 00:46
  • I should probably do `d.update(dict_tmp)` to use shelve in a better way. Thank you @user2357112, I think you've answered my question. –  Jul 27 '13 at 01:01

0 Answers0