namedtuple pickling fails when variable name doesn't match typename

Question

The python code below fails with the error pickle.PicklingError: Can't pickle <class '__main__.SpecialName'>: it's not found as __main__.SpecialName

import pickle
from collections import namedtuple

different_SpecialName = namedtuple('SpecialName', 'foo bar')
def save():
    foo = different_SpecialName(1, 2)
    with open('foosave.pkl', 'w') as f:
        pickle.dump(foo, f)

if __name__ == '__main__':
    save()

This seems like bad behaviour of the pickle module, as it depends on the correctness of a variable name. Changing different_SpecialName to SpecialName and re-running the code allows it to complete successfully. Changing the code to the below, where a variable with SpecialName is instantiated to be the same value as different_SpecialName, also lets the code run successfully

import pickle
from collections import namedtuple

different_SpecialName = namedtuple('SpecialName', 'foo bar')

## create new variable with 'correct' name
SpecialName = different_SpecialName

def save():
    # foo = different_SpecialName(1, 2)
    foo = SpecialName(1, 2)
    with open('foosave.pkl', 'w') as f:
        pickle.dump(foo, f)

if __name__ == '__main__':
    save()

My questions: is this fundamentally a pickle (and cPickle) bug? It seems like pickle shouldn't be looking up the class definition by using the name of the variable (although, I'm not sure what else it could do). Or, instead, is this an issue with the namedtuple API? I browsed the namedtuple documentation and couldn't find anything that explicitly told me to name my namedtuple variables the same as my typename argument (the first argument to the namedtuple() function)

score 1 · Accepted Answer · answered Sep 25 '15 at 00:35

1

It's not a bug. pickle requires that

the class definition must be importable and live in the same module as when the object was stored.

From the perspective of the namedtuple's __reduce__ method, the type name is SpecialName (that's what you passed it after all). So when unpickling, it will try to import the module it was declared in and look for SpecialName. But since you didn't save it as SpecialName, it can't find it.

Without resorting to namedtuples, you can produce the exact same problem with:

class Foo:
    pass
Bar = Foo
del Foo

and trying to pickle and unpickle a Bar(); under the hood, you've effectively done the same thing with your mismatched names for a namedtuple.

answered Sep 25 '15 at 00:35

ShadowRanger

143,180
12
188
271

great, so it's the `namedtuple` documentation failing to warn the user that mismatching names can result in errors. – eqzx Sep 25 '15 at 14:42
1

Well, every example given matches the `typename` argument with the name the result is assigned to, and the very first line notes that the class returned uses the `typename` argument. Adding a warning for every non-standard usage would clutter the docs quite a bit. – ShadowRanger Sep 26 '15 at 03:01
I disagree, it would take a brief one line to comment on the quite nonstandard requirement that the two names must match exactly. – eqzx Sep 26 '15 at 17:09

namedtuple pickling fails when variable name doesn't match typename

1 Answers1