48

i need to append to a pickle file (as i don't have the entire dictionary with me at one go). So for doing the same I have written the following code:

 import pickle
 p={}
 p[1]=2
 q={}
 q['a']=p
 p={}
 p[2]=0
 pickle.dump(q, open("save12.p","ab"))
 f={}
 f['b']=p
 pickle.dump(f,open("save12.p","ab"))

However, when I am loading the pickle file I don't find the value of dictionary f there?????

Can someone please suggest as to how should I go about appending in a pickle file???

Also databases like 'dbm' are not working for my need as i am working on windows

Jannat Arora
  • 2,759
  • 8
  • 44
  • 70
  • What makes you think that two appended pickle streams will somehow be magically accepted as one new object? If your data is too big to fit into memory, use a database (you have many choices, dbm ins't the only thing out there). – Chris Eberle Oct 06 '12 at 17:20
  • The [`ZODB`](http://www.zodb.org/) works fine on windows; it stores pickles too. – Martijn Pieters Oct 06 '12 at 17:20

2 Answers2

99

Pickle streams are entirely self-contained, and so unpickling will unpickle one object at a time.

Therefore, to unpickle multiple streams, you should repeatedly unpickle the file until you get an EOFError:

>>> f=open('a.p', 'wb')
>>> pickle.dump({1:2}, f)
>>> pickle.dump({3:4}, f)
>>> f.close()
>>> 
>>> f=open('a.p', 'rb')
>>> pickle.load(f)
{1: 2}
>>> pickle.load(f)
{3: 4}
>>> pickle.load(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
EOFError

so your unpickle code might look like

import pickle
objs = []
while 1:
    try:
        objs.append(pickle.load(f))
    except EOFError:
        break
nneonneo
  • 171,345
  • 36
  • 312
  • 383
30
#To append to a pickle file
import pickle

p={1:2}
q={3:4}
filename="picklefile"
with open(filename, 'a+') as fp:
    pickle.dump(p,fp)
    pickle.dump(q,fp)


#To load from pickle file
data = []
with open(filename, 'rb') as fr:
    try:
        while True:
            data.append(pickle.load(fr))
    except EOFError:
        pass
Rangoli Thakur
  • 335
  • 3
  • 4