12

Why can't I pickle a typing.NamedTuple while I can pickle a collections.namedtuple? How can I manage to do pickle a NamedTuple?

This code shows what I have tried so far:

from collections import namedtuple
from typing import NamedTuple

PersonTyping = NamedTuple('PersonTyping', [('firstname',str),('lastname',str)])
PersonCollections = namedtuple('PersonCollections', ['firstname','lastname'])

pt = PersonTyping("John","Smith")
pc = PersonCollections("John","Smith")


import pickle
import traceback

try:
    with open('personTyping.pkl', 'wb') as f:
        pickle.dump(pt, f)
except:
    traceback.print_exc()
try:
    with open('personCollections.pkl', 'wb') as f:
        pickle.dump(pc, f)
except:
    traceback.print_exc()

Output on the shell:

$ python3 prova.py 
Traceback (most recent call last):
  File "prova.py", line 16, in <module>
    pickle.dump(pt, f)
_pickle.PicklingError: Can't pickle <class 'typing.PersonTyping'>: attribute lookup PersonTyping on typing failed
$
Gandaro
  • 3,427
  • 1
  • 17
  • 19
marcotama
  • 1,991
  • 2
  • 19
  • 24

1 Answers1

6

It's a bug. I have opened a ticket on it: http://bugs.python.org/issue25665

The issue is that namedtuple function while creating the class sets its __module__ attribute by looking up __name__ attribute from the calling frame's globals. In this case the caller is typing.NamedTuple.

result.__module__ = _sys._getframe(1).f_globals.get('__name__', '__main__')

So, it ends up setting it up as 'typing' in this case.

>>> type(pt)
<class 'typing.PersonTyping'>  # this should be __main__.PersonTyping
>>> type(pc)
<class '__main__.PersonCollections'>
>>> import typing
>>> typing.NamedTuple.__globals__['__name__']
'typing'

Fix:

Instead of this the NamedTuple function should set it itself:

def NamedTuple(typename, fields):

    fields = [(n, t) for n, t in fields]
    cls = collections.namedtuple(typename, [n for n, t in fields])
    cls._field_types = dict(fields)
    try:
        cls.__module__ = sys._getframe(1).f_globals.get('__name__', '__main__')
    except (AttributeError, ValueError):
        pass
    return cls

For now you can also do:

PersonTyping = NamedTuple('PersonTyping', [('firstname',str),('lastname',str)])
PersonTyping.__module__ = __name__
Ashwini Chaudhary
  • 244,495
  • 58
  • 464
  • 504
  • Both your fixes work, thank you very much! Also, thanks for the explanation and for having already submit a patch! – marcotama Nov 19 '15 at 13:38
  • If the `AttributeError` or `ValueError` triggers, the resulting NamedTuple still will not be picklable, correct? You should mention that. – Ethan Furman Jan 13 '16 at 21:14