python object save and reload

Question

I've been working on a python program that basically creates 5 different types of objects that are in a hierarchy. For example my program might create 1 Region object that contains 2000 Column objects that contains 8000 Cell objects(4 Cells in each Column) where all the objects are interacting with each other based on video input.

Now, I want to be able to save all the objects states after the video input changes each of their states over a period of time. So my question is how can I save and reload thousands of objects in Python efficiently? Thanks in advance!

Efficiently, or quickly? – Ignacio Vazquez-Abrams Aug 17 '12 at 05:53 — Ignacio Vazquez-Abrams, Aug 17 '12 at 05:53

score 1 · Accepted Answer · answered Aug 17 '12 at 05:56

Not sure how efficient pickle is for large scales but I think what you're looking for is object serialization. But are you trying to 'refresh' the information in these objects or save and load them? Also read the section on 'Persistence of External Objects' since you will need to create an alphanumeric id that is associated with each object, for the relations/associations.

One totally hacky way could also be to json-ify the objects and store that. You would still need the alphanumeric id or some sort of usable identifier to associate each of the objects.

There's also `shelve` which @IgnacioVazquez-Abrams mentions in another SO thread: http://stackoverflow.com/a/12000115/1431750 — aneroid, Aug 17 '12 at 06:01

score 1 · Answer 2 · answered Aug 17 '12 at 07:05

1

Have you looked at Shelve, Pickle or cPickle? http://docs.python.org/release/2.5/lib/persistence.html

answered Aug 17 '12 at 07:05

Sachin

915
6
10

score 0 · Answer 3 · answered Aug 17 '12 at 06:37

I think you need to look into the ZODB.

The ZODB is an object database that uses pickle to serialize data, is very adept at handling hierarchies of objects, and if your objects use the included persistent.Persistent base-class, will detect and only save the objects that changed when you commit; e.g. there is no need to write out the whole hierarchy on every little change.

Included in the ZODB project is a package called BTrees, which are ZODB aware and make storing thousands of objects in one place efficient. Use these for your Region object to store the Columns. We use BTrees to store millions of datapoints at times.

python object save and reload

3 Answers3