How to populate shelf with existing dictionary

Question

Lets say I have a large, 100s of megabytes, dictionary that I want to make into an on disk shelve. I'm using pypar to utilize MPI to generate cleaned bits of a master list. What's the best way to achieve this? Example:

# much earlier
masterDict = shelve.open( 'aShelveFile' )
# .. . . . . 
# then we work out which parts of masterDict to keep
# and we put into aCleanDict
# then use mpi pypar to send the cleaned bits to the master rank
if pypar.rank() == 0:
  tempdict = {}
  for p in range(1,pypar.size()):
    tempdict.append(pypar.receive(p))
  for l1 in tempdict:
    for l2 in l1:
      realDict.append(l2)
  for p in range(1,pypar.size()):
    pypar.send(realDict,p)

  # now realDict has all the cleaned bits
  # which we send to the other hosts
else:
  pypar.send(aCleanDict, 0 )
  aCleanDict = pypar.receive( 0 )

# now to replace masterDict  with aCleanDict
# note: cannot send shelve dictonaries using pypar

# insert stackover flow code here.

I added a simple example. Not sure if this is what you need though. If not, you may want to clarify the question a bit. — juanchopanza, Jul 10 '11 at 10:49

juanchopanza · Accepted Answer · 2011-07-10T11:27:43.147

Here you shelve a simple dictionary and make it accessibly via key myDict:

import shelve
myDict = {"a" : 1, "b" : 2}
myShelvedDict = shelve.open("my_shelved_dictionary.db")
myShelvedDict["myDict"] = myDict

Note that the contents of the dictionary must be pickleable, as for anything that is to be shelved.

If you want to replicate the structure of the dictionary in the shelve, i.e. not have a myDict key but the keys of the dictionary directly as keys of the shelf, you can use the update method of shelve:

myShelvedDict.update(myDict)

The shelve interface has a large overlap with the dict one.

How to populate shelf with existing dictionary

1 Answers1