Python dill: Pickle namedtuple doesnt seem to work

Question

I have namedtuple inside a class. When pickling using dill, it complains the classic issue of not being able to find the namedtuple object at top module.

import dill as pickle    
class NNRTMParse(object):

    def __init__(self,logfile)):
        .
        .
        .
        .

        self.TM = namedtuple('TM',tmeas_fields)
        #print self.TM
        CFH = namedtuple('CFH',cfhdr_fields)
        PM = namedtuple('PM',pmeas_fields2)
        print PM

This is default assignment for self.TM and others, and I dynamically assign namedtuple as I parse the log.

But pickling :

 if __name__ == "__main__":

    filename = 'dbggen_rx_loc_2-llh_rtm_lla_out_20160929_130711_day2_4381_JN2_SN64_rtmproc_2M5M.txt'
    N = NNRTMParse(filename)
    N.parse()
    N.get_rx_loc('oak484_bora-llh')

    filehandler = open("NNRTMParse_JB2-SN052.obj","wb")
    pickle.dump(N,filehandler)
    filehandler.close()






Traceback (most recent call last):
  File "C:/NN_Hardware/spos_proc/NNRTMParse.py", line 937, in <module>
    pickle.dump(N,filehandler)
  File "C:\Python27\lib\site-packages\dill\dill.py", line 236, in dump
    pik.dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\site-packages\dill\dill.py", line 1189, in save_type
    StockPickler.save_global(pickler, obj)
  File "C:\Python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <class '__main__.TM'>: it's not found as __main__.TM

I am not sure if there is any way out. Else would have to route thru the path just to nit-pick the dataframes to pickle, which I hate to do.

Any help is very appreciated.

Hi, I'm the `dill` author. I think the issue might be something along these lines: https://github.com/uqfoundation/dill/issues/132. With `dill`, a `namedtuple` pickles pretty well, as long as it is not named "badly". Basically, if you name it as shown in the above issue, then use a pointer reference for what you want, that might just "work". The key is having the zeroth argument the same as the `__name__`. — Mike McKerns, Oct 07 '16 at 07:22
@MikeMcKerns thanks for the reply. I am not sure - I dont have the "bad" naming. I tried - TM = namedtuple('TM',tmeas_fields) TM.__name__ = "TM" print TM output: . pickle.PicklingError: Can't pickle : it's not found as __main__.TM — rrkarts, Oct 08 '16 at 17:09

score 0 · Answer 1 · answered Oct 09 '16 at 09:27

0

The issue is the self.TM namedtuple. If you don't use the namedtuple as a class attribute, then your class should pickle.

# file: xxx.py
from collections import namedtuple

class NNRTMParse(object):

  def __init__(self):
    TM = namedtuple("TM", 'a')
    CFH = namedtuple("CFH", 'b')
    print CFH

It should work like this:

>>> from xxx import *
>>> a = NNRTMParse()
<class 'xxx.CFH'>
>>> import dill
>>> dill.copy(a)
<xxx.NNRTMParse object at 0x107da3350>

However, as soon as you try to use the namedtuple as a attribute, it fails with the error you are seeing.

I'd suggest adding a __reduce__ method (https://docs.python.org/3/library/pickle.html#object.reduce) to tell pickle how to serialize the state of the class instance.

answered Oct 09 '16 at 09:27

Mike McKerns

33,715
8
119
139

Even without the _self.TM_ , just like my earlier comment, I tried precisely what you have suggested here, i.e. not make _namedtuple_ as an attribute. But I still get same error. On the alternate suggestion of yours, if I would use say _self.TM_ , and put _reduce_(). Not sure if I have this correct but still get smae error. def __reduce__(self): return (self.__class__, (self.TM, self.PM, self.SPOSX, self.CFH )) – rrkarts Oct 10 '16 at 18:35
Hmm, it'd help to see a full reduced version of your code then -- one which still produces the error… because my version above works until you use `self.TM`. – Mike McKerns Oct 10 '16 at 20:51

Python dill: Pickle namedtuple doesnt seem to work

1 Answers1