What is the best way to save a complex dictionary to file?

Question

I have a dictionary that has tuple keys and numpy array values. I tried saving it using h5 and pickle but I get error messages. what is the best way to save this object to file?

import numpy as np
from collections import defaultdict
Q =defaultdict(lambda: np.zeros(2))
Q[(1,2,False)] = np.array([1,2])
Q[(1,3,True)] = np.array([3,4])

>>> Q
defaultdict(<function <lambda> at 0x10c51ce18>, {(1, 2, False): array([1, 2]), (1, 3, True): array([3, 4])})

np.save traceback:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-99-a071e1561501> in <module>()
----> 1 np.save('Q.npy', Q)

~/anaconda3_420/lib/python3.5/site-packages/numpy/lib/npyio.py in save(file, arr, allow_pickle, fix_imports)
    509         arr = np.asanyarray(arr)
    510         format.write_array(fid, arr, allow_pickle=allow_pickle,
--> 511                            pickle_kwargs=pickle_kwargs)
    512     finally:
    513         if own_fid:

~/anaconda3_420/lib/python3.5/site-packages/numpy/lib/format.py in write_array(fp, array, version, allow_pickle, pickle_kwargs)
    584         if pickle_kwargs is None:
    585             pickle_kwargs = {}
--> 586         pickle.dump(array, fp, protocol=2, **pickle_kwargs)
    587     elif array.flags.f_contiguous and not array.flags.c_contiguous:
    588         if isfileobj(fp):

AttributeError: Can't pickle local object 'mc_control_epsilon_greedy.<locals>.<lambda>'

Answer 1

How about saving it as a plain dictionary? You don't need the defaultdict behavior during saving.

In [126]: from collections import defaultdict
In [127]: Q =defaultdict(lambda: np.zeros(2))
     ...: Q[(1,2,False)] = np.array([1,2])
     ...: Q[(1,3,True)] = np.array([3,4])
     ...: Q[(3,4,False)]
     ...: 
Out[127]: array([0., 0.])
In [128]: Q
Out[128]: 
defaultdict(<function __main__.<lambda>>,
            {(1, 2, False): array([1, 2]),
             (1, 3, True): array([3, 4]),
             (3, 4, False): array([0., 0.])})

We can pull it out of the defaultdict wrapping with:

In [130]: dict(Q)
Out[130]: 
{(1, 2, False): array([1, 2]),
 (1, 3, True): array([3, 4]),
 (3, 4, False): array([0., 0.])}

Then we can pickle it (I'm using np.save as a pickle shortcut)

In [131]: np.save('stack49963862', np.array(dict(Q)))

load gives an object array containing this dictionary:

In [132]: P = np.load('stack49963862.npy')
In [133]: P
Out[133]: 
array({(1, 2, False): array([1, 2]), (1, 3, True): array([3, 4]), (3, 4, False): array([0., 0.])},
      dtype=object)

In [138]: P.item()
Out[138]: 
{(1, 2, False): array([1, 2]),
 (1, 3, True): array([3, 4]),
 (3, 4, False): array([0., 0.])}

We can easily recreate the defaultdict with an update:

In [134]: Q1 =defaultdict(lambda: np.zeros(2))
In [139]: Q1.update(P.item())
In [140]: Q1
Out[140]: 
defaultdict(<function __main__.<lambda>>,
            {(1, 2, False): array([1, 2]),
             (1, 3, True): array([3, 4]),
             (3, 4, False): array([0., 0.])})

Answer 2

I don't see any problems using pickle

import pickle
import numpy as np

x = {(1,2,False): np.array([1,4]), (1,3,False): np.array([4,5])}

with open('filename.pickle', 'wb') as handle:
    pickle.dump(x, handle, protocol=pickle.HIGHEST_PROTOCOL)
with open('filename.pickle', 'rb') as handle:
    y = pickle.load(handle)

print x
print y

---

After the edit:

What you actually have is a lambda, and that cannot be pickled by default. You need to install dill and import it for it to work (see this answer)

This is how it should look like:

import pickle
import numpy as np
from collections import defaultdict
import dill # doesn't come with default anaconda. Install with "conda install dill"

x = defaultdict(lambda: np.zeros(2))
with open('filename.pickle', 'wb') as handle:
    pickle.dump(x, handle, protocol=pickle.HIGHEST_PROTOCOL)
with open('filename.pickle', 'rb') as handle:
    y = pickle.load(handle)

print x
print y

Output:

# no errors :-)
defaultdict(<function <lambda> at 0x000000000CD0C898>, {})
defaultdict(<function <lambda> at 0x0000000002614C88>, {})

---

OP's solution: your edited solution still generated the same error for me but this works fine:

import pickle
import dill
dill_file = open("Q.pickle", "wb")
dill_file.write(dill.dumps(Q))
dill_file.close()

On my machine (Win 8.1 64 bit, using Spyder), I had no errors when using simple dill.