0

I'm using pyro5, and I want to make a remote object function as a worker class, where a client can ask it to perform various tasks by sending it functions to execute.

For this to work, I need to serialize functions, but the serializers available through Pyro5 does not support serializing functions.

Instead, I intend to use Dill, and somehow send the already serialized function through Pyro5.

However, when trying to serialize a function, I receive errors which say that I can't serialize X, where X is the enclosing class where the function lives. How can I prevent this from happening?

Actually, I was not able to reproduce this in a small script with a class and a function. I am running the dill serializer in a Qt app, not sure if that's the problem?

Alternatively, does anyone have a better idea on how to approach this?

Ken White
  • 123,280
  • 14
  • 225
  • 444
FilterFeeder
  • 3
  • 3

1 Answers1

0

I'm the dill author. There are several ways to approach this that may work for you. It's hard to tell without more information on your class in question, so I'll have to make assumptions.

If assume you are working with a class that you are importing from an installed package, then you can do something like this:

>>> # Mock a unserializable builtin class "Foo"
>>> import ctypes
>>> PyObjectType = ctypes.py_object(lambda :None)
>>> class Foo(PyObjectType.__class__):
...   def __init__(self, x):
...     self.x = x
...   def bar(self, y):
...     return y + self.x
... 
>>> f = Foo(1)
>>> f.bar(2)
3
>>> # serializing the method fails
>>> # dill.dumps(f.bar) #FAILS
>>> # so we create a wrapping function
>>> bar = lambda x:f.bar(x)
>>> 
>>> import dill
>>> dill.dumps(bar)
b'\x80\x03cdill._dill\n_create_function\nq\x00(cdill._dill\n_create_code\nq\x01(K\x01K\x00K\x01K\x03KCC\nt\x00\xa0\x01|\x00\xa1\x01S\x00q\x02N\x85q\x03X\x01\x00\x00\x00fq\x04X\x03\x00\x00\x00barq\x05\x86q\x06X\x01\x00\x00\x00xq\x07\x85q\x08X\x07\x00\x00\x00<stdin>q\tX\x08\x00\x00\x00<lambda>q\nK\x01C\x00q\x0b))tq\x0cRq\rc__builtin__\n__main__\nh\nNNtq\x0eRq\x0f}q\x10}q\x11X\x0f\x00\x00\x00__annotations__q\x12}q\x13s\x86q\x14b.'

This works, as long as the class is available for import wherever you are going to use dill.loads to restore the function.

Or if you need to ship the class as well, you could also create a wrapping class (i.e. a derived class) that includes a __reduce__ method which informs dill how to serialize the class. https://docs.python.org/3/library/pickle.html#object.__reduce__

There are other approaches as well, but it would help to have more information on what you are dealing with.

Mike McKerns
  • 33,715
  • 8
  • 119
  • 139
  • Thank you, I highly appreciate you personally stepping in to support your package users. I was probably very unclear, but I didn't want to serialize the function as a method, rather a function, and I found a way to fish out the function from the method. Either way, I have problems de-serializing the function. I receive the error "ImportError: cannot import name 'Annotated' from 'typing' (/usr/lib/python3.8/typing.py)" when calling dill.loads(). I am in fact doing this on a remote object trough Pyro5, not sure if that is the root cause of the issue. – FilterFeeder Sep 11 '22 at 20:05
  • As a note, I tried to use the remote object to return the serialized data, and I successfully de-serialize the function on the local machine (as opposed to on the remote machine). This suggests that it's not Pyro5 that is messing with the serialized data, but that I'm doing something wrong when de-serializing the function on the remote machine. – FilterFeeder Sep 11 '22 at 20:35
  • @FilterFeeder: it's rare, but serialization can sometimes fail on `load`. It shouldn't matter to the serializer whether you are serializing the method, or extracting the function (using `__func__`, I'm assuming) from the method. Maybe you can open a ticket on the `dill` GitHub page, as that might be a better interactive format to get you an answer to this issue. – Mike McKerns Sep 12 '22 at 13:36