4

I use pickle and dill for follow lambda function and work fine :

import dill
import pickle

f = lambda x,y: x+y
s = pickle.dumps(f)

or even when used in class, for example:

file

foo.py

class Foo(object):
    def __init__(self):
        self.f = lambda x, y: x+y

file

test.py

import dill
import pickle
from foo import Foo

f = Foo()
s = pickle.dumps(f)  # or  s = dill.dumps(f)

but when build same file with format .pyx (foo.pyx) using cython, can't serialize with dill, pickle or cpickle, get this error :

Traceback (most recent call last): File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2878, in run_cod exec(code_obj, self.user_global_ns, self.user_ns) File "", line 1, in a = pickle.dumps(c) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 1380, in dumps Pickler(file, protocol).dump(obj) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 224, in dump self.save(obj) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 425, in save_reduce save(state) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/site-packages/dill/_dill.py", line 912, in save_module_dict StockPickler.save_dict(pickler, obj) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 655, in save_dict self._batch_setitems(obj.iteritems()) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 669, in _batch_setitems save(v) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 317, in save self.save_global(obj, rv) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 754, in save_global (obj, module, name)) PicklingError: Can't pickle . at 0x7f9ab1ff07d0>: it's not found as foo.lambda

setup.py file for build cython

setup.py

from distutils.core import setup
from Cython.Build import cythonize

setup(ext_modules=cythonize("foo.pyx"))

then run in terminal:

python setup.py build_ext --inplace

Is there a way ?

Amin
  • 138
  • 1
  • 8
  • `pickle` is already complied code. What do you how to accomplish with `cython`? – hpaulj Apr 10 '20 at 15:14
  • @hpaulj i just compiled my class (foo.pyx) not file contain pickle (test.py) and when i dont use lambda in foo.pyx class, this is work fine just get error when use lambda. – Amin Apr 10 '20 at 15:43
  • 1
    @kiba Realistically this isn't going to work. A Cython lambda isn't the same as a Python lambda, and I don't think any effort has been made to make it pickle-able. Generally lambdas will have some closure variables and I imagine those are especially non-pickleable in Cython. – DavidW Apr 10 '20 at 16:56

2 Answers2

2

I'm the dill author. Expanding on what @DavidW says in the comments -- I believe there are (currently) no known serializers that can pickle cython lambdas, or the vast majority of cython code. Indeed, it is much more difficult for python serializers to be able to pickle objects with C-extensions unless the authors of the C-extension code specifically build serialization instructions (as did numpy and pandas). In that vein... instead of a lambda, you could build a class with a __call__ method, so it acts like a function... and then add one or more of the pickle methods (__reduce__, __getstate__, __setstate__, or something similar)... and then you should be able to pickle instances of your class. It's a bit of work, but since this path has been used to pickle classes written in C++ -- I believe you should be able to get it to work for cython-built classes.

Mike McKerns
  • 33,715
  • 8
  • 119
  • 139
0

In this code

import dill
import pickle

f = lambda x,y: x+y
s = pickle.dumps(f)

f is a function, But in another code

import dill
import pickle
from foo import Foo

f = Foo()
s = pickle.dumps(f)  
# or  
s = dill.dumps(f)

f is a class

Michael Szczesny
  • 4,911
  • 5
  • 15
  • 32
stack12138
  • 1