TypeError: can't pickle Struct objects

Question

I am the maintainer of the python package Construct and I seek help in making this library picklable. Someone came to me and asked for it to be cloudpickle-able. Unfortunately the classes I have are not pickle-able nor cloudpickle-able nor dill-able. Please help.

The relevant ticket is: https://github.com/construct/construct/issues/894

$ python3
Python 3.6.9 (default, Oct  8 2020, 12:12:24) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cloudpickle
>>> import construct
>>> cloudpickle.dumps(construct.Byte)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps
    cp.dump(obj)
  File "/usr/local/lib/python3.6/dist-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/usr/lib/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/usr/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.6/pickle.py", line 634, in save_reduce
    save(state)
  File "/usr/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/lib/python3.6/pickle.py", line 496, in save
    rv = reduce(self.proto)
TypeError: can't pickle Struct objects

Same error goes for dill. Pickle module produces a one-line error.

I'm the `dill` author. As a package maintainer, you have the option of adding class methods that instruct python how to serialize your class. See: https://docs.python.org/3/library/pickle.html#pickling-class-instances. This can enable you to pickle classes that have unpicklable items in them. — Mike McKerns, Feb 04 '21 at 14:49

ArekBulski · Accepted Answer · 2021-02-03T11:16:52.820

Solved: The error gave me one clue. Byte class object which I tried to pickle is a FormatField and has nothing to do with the Struct class. Only after few hours of thinking about it, it occured to me that Struct refers to struct.Struct and not construct.Struct. After getting rid of it, it serializes properly.

Empty construct.Struct class object serializes without issues.

Offending code:

class FormatField(Construct):
    def __init__(self, endianity, format):
        if endianity not in list("=<>"):
            raise FormatFieldError("endianity must be like: = < >", endianity)
        if format not in list("fdBHLQbhlqe?"):
            raise FormatFieldError("format must be like: f d B H L Q b h l q e ?", format)

        super().__init__()
        self.fmtstr = endianity+format
        self.length = struct.calcsize(endianity+format)
        self.packer = struct.Struct(endianity+format) # <---- culprit

TypeError: can't pickle Struct objects

1 Answers1