Pickling Cython decorated function results in PicklingError

Question

I have the following code:

def decorator(func):

    @functools.wraps(func)
    def other_func():
        print('other func')

    return other_func

@decorator
def func():
    pass

If I try to pickle func everything works. However if I compile the module as a Cython extension it fails. Here is the error:

>>>> pickle.dumps(module.func)

PicklingError: Can't pickle <cyfunction decorator.<locals>.other_func at 0x102a45a58>: attribute lookup other_func on module failed

The same happens if I use dill instead of pickle.

Do you know how to fix it?

Have you tried specifying a `return` for `def other_func():`? — ycx, Dec 14 '18 at 09:16
Yes, it is the same actually. Why do you think it could help? — stefano1, Dec 14 '18 at 09:27
I can't test this at the minute but: is this specifically a Cython problem? Would this code work with pure Python? (I don't think it should be possible to pickle functions defined within other functions, but I think dill should work. I could be wrong though...) — DavidW, Dec 14 '18 at 11:05
Yes, it works as pure Python. It is not possible to pickle without adding `@functools.wraps(func)`, but with it it works. `dill` works also without this, but fails in the Cython case. — stefano1, Dec 14 '18 at 11:09

Dunes · Answer 1 · 2018-12-14T13:22:32.003

I don't think there is anything you can really do here. It looks like a possible bug in Cython. But there might be a good reason for why Cython does what it does that I don't know about.

The problem arises because Cython functions are exposed as builtin functions in Python land (eg. map, all, etc.). These functions cannot have their name attributes changed. However, Cython attempts to make its functions more like pure Python functions, and so provides for the ability for several of their attributes to modified. However, the Cython functions also implement __reduce__ which customises how objects are serialised by pickle. It looks like this function does think the name of the function object can be changed and so ignores these values and uses the name of the internal PyCFunction struct that is being wrapped (github blob).

Best thing you can do is file a bug report. You might be able to create a thin wrapper than enables your function to be serialised, but this will add overhead when the function is called.

Customising Pickle

You can use the persistent_id feature of the Pickler and Unpickler to override the custom implementation that Cython has provided. Below is how to customise pickling for specific types/objects. It's done with a pure python function, but you can easily change it to deal with Cython functions.

import pickle
from importlib import import_module
from io import BytesIO

# example using pure python
class NoPickle:
    def __init__(self, name):
        # emulating a function set of attributes needed to pickle
        self.__module__ = __name__
        self.__qualname__ = name

    def __reduce__(self):
        # cannot pickle this object
        raise Exception


my_object = NoPickle('my_object')

# pickle.dumps(obj) # error!

# use persistent_id/load to help dump/load cython functions

class CustomPickler(pickle.Pickler):
    def persistent_id(self, obj):
        if isinstance(obj, NoPickle):
            # replace with NoPickle with type(module.func) to get the correct type
            # alternatively you might want to include a simple cython function 
            # in the same module to make it easier to get the write type.
            return "CythonFunc" , obj.__module__, obj.__qualname__
        else:
            # else return None to pickle the object as normal
            return None

class CustomUnpickler(pickle.Unpickler):
    def persistent_load(self, pid):
        if pid[0] == "CythonFunc":
            _, mod_name, func_name = pid
            return getattr(import_module(mod_name), func_name)
        else:
            raise pickle.UnpicklingError('unsupported pid')

bytes_ = BytesIO()
CustomPickler(bytes_).dump(my_object)

bytes_.seek(0)
obj = CustomUnpickler(bytes_).load()

assert obj is my_object

I see, thanks a lot! I will fill the bug report. Do you know or have references about possible wrappers to write? — stefano1, Dec 14 '18 at 12:47
I've added a code sample to show how you could customise your pickling. I don't have cython on this machine, so I haven't fully tested it. But I don't see any reason for it not to work. — Dunes, Dec 14 '18 at 13:24

Pickling Cython decorated function results in PicklingError

1 Answers1

Customising Pickle