0

I have 26 .pkl files at hand, from dict_a.pkl to dict_z.pkl. I want to load them into the memory, so I can compare elements start with a with variable loaded from dict_a.pkl. The reason I want to do this is that each file is really large, if I put them all in one big file, it will be too large to digest. If I load files in an ad hoc style, then it would constantly read disk.

import string
alphabet = string.lowercase

for alpha in alphabet:
   ff = 'dict_'+alpha+'.pkl'
   with open(ff, 'r') as tt:
       temp = cPickle.load(tt)

How can I replace temp variable with dict_a, dict_b in the loop, so I after the loop, I can directly use the variable dict_a to dict_z.

Sean
  • 1,161
  • 1
  • 13
  • 24
  • 1
    You could have a `dict` of keys `'dict_a'`, `dict_b`, ... with the corresponding values being the pickle objects – mechanical_meat Aug 21 '14 at 05:44
  • One trick you might like: you can save multiple pickles to a single file then load them one by one. You don't need 26 files. – John Zwinck Aug 21 '14 at 05:50
  • Thanks, @bernie. My pickled stuffs are list with more than one dimension. When indexing them, the code would be so unreadable. I will try to figure out some other way. – Sean Aug 21 '14 at 06:51

1 Answers1

0

You can use the glob module to load the pickled files and then work on them one by one.

    import glob
    import cPickle as pickle
    picklefiles = glob.glob('*.pkl')
    for pklfile in picklefiles:
       with open(pklfile,'rb') as f:
           pklfile = pickle.load(f)

Now you have 26 dictionaries with their corresponding file names.

Edit : you're right @Sean i messed up the third line it should be picklefiles instead of pklfiles. I've updated the code snippet

Edit 2: picklefiles is a list containing all the file names with .pkl extension. In this case 

picklefiles = [  'dict_a.pkl', 'dict_b.pkl', ........ 'dict_z.pkl' ]

In the for loop i'm opening each file and loading the dictionary into a dictionary with the corresponding file name.

Ex :

( this is a variable of type dictionary ) dict_a.pkl = { values from dict_a.pkl }

HaseebR7
  • 447
  • 4
  • 11
  • I try to run your code. The third line should be ('*.pkl'), if I am correct. – Sean Aug 21 '14 at 06:29
  • I try to run your code. The third line should be ('*.pkl'), if I am correct. `pklfiles` is just a list containing all the file names of pkl files, like `dict_x.pkl`. From the code, I suppose it will output variables like `dict_x.pkl`. But this definitely is not a variable name. After running the code, i only got `pklfile` as a variable containing the content of the last pkl file. – Sean Aug 21 '14 at 06:36
  • Maybe I didn't run it right, after running the code, I only got one variable `pklfile` containing the content of the last pkl file. – Sean Aug 21 '14 at 06:41
  • the third line should be `picklefiles = glob.glob('*.pkl')`. And the result is `picklefiles`, right? – Sean Aug 21 '14 at 07:08
  • So, actually in `for` loop. You use pickled file to replace the original string in picklefiles? – Sean Aug 21 '14 at 07:10
  • @Sean I didn't know i had enough rights to add comments. LOL. Check edit 2. – HaseebR7 Aug 21 '14 at 07:29
  • I try to print picklefiles, the result is `['list_x.pkl', 'list_y.pkl']`. Oh, I forgot to mention, I only use two files as toy. – Sean Aug 21 '14 at 07:33
  • @luzlage also when to print `list_x.pkl` or `list_x`, the error says it is not defined. I understand the code and your comment, but still it is not quite right. – Sean Aug 21 '14 at 07:36