3

I've been using Shelve as a document store. The key is a stringified integer and the value is just some html in a string. Unfortunately my script ended up putting so many entries in the db that errors occurred (I don't have the exact ones to hand). The db is about 36GB in size and now when I load it and then try and iterate on the keys or anything like that I get the following error...

import shelve

db = shelve.open("my.shelf")
ks = db.keys()
for k in ks: print(k)
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/_collections_abc.py", line 482, in __iter__
    yield from self._mapping
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/shelve.py", line 95, in __iter__
    for k in self.dict.keys():
SystemError: Negative size passed to PyBytes_FromStringAndSize

>>> list(ks.__dict__.values())[0].dict
<_dbm.dbm object at 0x10037ef90>
>>> help(list(ks.__dict__.values())[0].dict)

>>> list(ks.__dict__.values())[0].dict.keys()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: Negative size passed to PyBytes_FromStringAndSize

This is on OSX Yosemite. Python 3.4

Any way to repair this or get the keys and values out for placement in a more appropriate store?

Johnny
  • 31
  • 2
  • It looks like your size is so large it overcame the integer limit, hence went into negative numbers. One suggestion I have is to do this: `for k in ks: yield k` -- using the `yield` keyword creates a generator which only returns a value when it is in use. http://simeonvisser.com/posts/python-3-using-yield-from-in-generators-part-1.html – NuclearPeon Mar 30 '15 at 17:20

1 Answers1

0

You must know key names whose corresponding objects were successfully stored.

Because a failure at saving data object would corrupt your db.

Then,

db = shelve.open("my.shelf") for key in list_of_successfully_saved_obj's_name: val = db.get(key) ...

displayname
  • 663
  • 8
  • 9