17

I am still pretty new to Python and even newer to pickling. I have a class Vertex(ScatterLayout) with a __getnewargs__():

def __getnewargs__(self):
    return (self.pos, self.size, self.idea.text)

My understanding is that this will cause the pickle to pickle the object from __getnewargs__() rather than the object's dictionary.

The pickle is called in the following method (in a different class MindMapApp(App)):

def save(self):
    vertices = self.mindmap.get_vertices()
    edges = self.mindmap.get_edges()

    output = open('mindmap.pkl', 'wb')

    #pickle.dump(edges, output, pickle.HIGHEST_PROTOCOL)
    pickle.dump(vertices, output, pickle.HIGHEST_PROTOCOL)

    output.close()

When I call the save() method I get the following error:

pickle.PicklingError: Can't pickle <type 'weakref'>: it's not found as __builtin__.weakref

What am I missing or not understanding? I have also tried implementing the __getstate__() / __setstate__(state) combination, with the same result.

David Poxon
  • 2,435
  • 4
  • 23
  • 44
  • do you mean class has __getnewargs__ and that will cause error when do pickle.dump(a), if suppose class A and a = A()? Because i try it and no error return, could you give more details about problem? like how your class defined – linpingta May 14 '14 at 03:48
  • I think the issue is I need to knwo two things: a) how to handle weakrefs when pickling a class; b) why a weakref is being created when the items that are pickled a returned by `__getnewargs__()`. – David Poxon May 14 '14 at 03:57
  • You're not pickling the class instance (i.e. `self`), you're pickling `vertices`. Which is, presumably, a `weakref`. – roippi May 14 '14 at 04:05
  • @roippi How do I resolve this? I tried pickling each instance by looping through the `vertices` list and got the same error. – David Poxon May 14 '14 at 04:24
  • I converted the list to a dictionary. Didn't help. – David Poxon May 14 '14 at 04:50

2 Answers2

17

You definitely can serialize a weakref, and you can serialize a dict and a list. Basically, it matters what the objects contain. If the dict or list contains any unpicklable items, then the pickling will fail. If you want to pickle a weakref, you have to use dill and not pickle. dill extends pickle to include objects that are otherwise unpicklable with pickle. However, note that with dill, an unpickled weakref will deserialize as dead references.

>>> import dill
>>> import weakref
>>> dill.loads(dill.dumps(weakref.WeakKeyDictionary()))
<WeakKeyDictionary at 4528979192>
>>> dill.loads(dill.dumps(weakref.WeakValueDictionary()))
<WeakValueDictionary at 4528976888>
>>> class _class:
...   def _method(self):
...     pass
... 
>>> _instance = _class()
>>> dill.loads(dill.dumps(weakref.ref(_instance)))
<weakref at 0x10d748940; dead>
>>> dill.loads(dill.dumps(weakref.ref(_class())))
<weakref at 0x10e246a48; dead>
>>> dill.loads(dill.dumps(weakref.proxy(_instance)))
<weakproxy at 0x10e246b50 to NoneType at 0x10d481598>
>>> dill.loads(dill.dumps(weakref.proxy(_class())))
<weakproxy at 0x10e246ba8 to NoneType at 0x10d481598>
Mike McKerns
  • 33,715
  • 8
  • 119
  • 139
  • hey using `dill` how can you store it in a ```.pkl``` file? – nasc Sep 17 '21 at 05:14
  • 1
    you mean like with `dill.dump(weakref, open('file.pkl', 'wb'))`? – Mike McKerns Sep 17 '21 at 13:04
  • oh cool, thanks dude – nasc Sep 17 '21 at 15:02
  • hey @Mike McKerns it still doesn't work i get this error ```TypeError: cannot pickle 'tensorflow.python.client._pywrap_tf_session.TF_Operation' object``` – nasc Sep 17 '21 at 15:33
  • Test cases I tried seem to work. You should submit a GitHub issue for dill. The comments on a SO post is not a good place to debug issues. – Mike McKerns Sep 17 '21 at 19:15
  • Sorry but I have to -1 for saying "you can pickle a weakref" unless someone can explain to me why using `dill` should be called "pickling", or why "pickling" should not specifically mean using `pickle`. Because to me that seems seriously misleading - not just some pedantic terminology nit that doesn't matter, but something liable to cause real problems from confusion/conflation/misconception. – mtraceur Jun 11 '22 at 20:02
  • In the python community "to pickle" and "to serialize" are pretty synonymous. Note also that "pickle" is commonly used as a verb. The primary mechanism for serializing an object in python is pickling (marshaling is much more rare). By saying "pickling" people can understand that `dill` extends the STL `pickle`... and thus `dill` is using the `pickle` protocol and `pickle` itself to serialize the weakref. Thus, it's said that `dill` can pickle a weakref. So, I'd argue that it's not misleading, it's actually technically accurate. – Mike McKerns Jun 12 '22 at 10:19
  • @MikeMcKerns I hear you and those are all true facts, I just think that's retroactively rationalizing metonymous usage as technically correct when really it's just as much precision-losing linguistic drift: sure, people have come to use "pickle" as a verb to mean the more general concept "serialize" - *because* a common and built-in serialization method is called "pickling" - but I don't think this is making things better for us, the actual benefit is tiny and in fact I suspect a lot of generalized usage of "pickle" is due to people who don't learn or value the distinction. – mtraceur Jun 13 '22 at 05:24
  • @MikeMcKerns Imagine you came across a JavaScript developer saying "oh just JSON it" when they really mean "serialize it", or you get a bug report saying "I'm having a bug when I JSONify this object returned by your library" and then you spend time trying to figure out what they're talking about and why and JSON serialization works fine in all your testing and then it turns out they meant some other serialization. But I notice that `dill` itself uses "pickle" like this in its readme, and it seems `dill` is wire-compatible for all objects that `pickle` can handle, so I'll take back my -1. – mtraceur Jun 13 '22 at 05:28
  • (Or I would take back my -1, but I guess enough time has passed so now stackoverflow locked my vote in until any edit happens to the post. But let the record show I was willing.) – mtraceur Jun 13 '22 at 05:29
  • @mtraceur: no worries. I'll clarify the language in the post a bit. – Mike McKerns Jun 13 '22 at 09:41
3

I worked around this by switching between weak/strong reference in __getstate__/__setstate__:

class DBObject(object):
    def __getstate__(self):
        s = self.__dict__.copy()
        s['_db'] = s['_db']()
        return s

    def __setstate__(self, state):
        self.__dict__ = state.copy()
        self.__dict__['_db'] = weakref.ref(self.__dict__['_db'])
kjam
  • 809
  • 1
  • 7
  • 17
  • 1
    the weak reference created in setstate will be a dead one, since the self.dict['_db'] in setstate is not existing anymore after exit the setstate scope, which means that the weak_reference() is returning None – Zhenshan Jin Mar 31 '20 at 00:46