What's different between these 2 partial function objects?

Question

Precondition

I have these definitions:

def add(x,y): return (lambda x,y: x+y)(x,y)
def call(f,x,y): return f(x,y)

and these import aliases:

from multiprocessing import Pool as P;
from functools import partial as p;

Then I run this:

P(2).map(p(add,1),[2,3])

I can get a result: [3, 4]

But if I run this:

P(2).map(p(p(call,lambda x,y: x+y),1),[2,3])

it produces an error message.

In Python version 2.7.5:

Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 342, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

(and then I have to press ^Z then kill %1 to close it)

In Python version 3.6.8:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
    put(task)
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 210, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib64/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7f25b741cd90>: attribute lookup <lambda> on __main__ failed

(better than python2: the REPL will give me the >>> again ...)

I've tried these, both are okay:

run: p(add,1)(3) , get: 4
run: p(p(call,lambda x,y: x+y),1)(3) , get: 4

Doubt

Now, I have my question:

I think they are same thing:

(x,y) -> (lambda x,y: x+y)(x,y)
partial((f,x,y) -> f(x,y) , lambda x,y: x+y)

Clearly they are in fact different. But how, and why?

While using multiprocessing, you cannot use the internal / nested functions because pickling does not support internal function objects. In first case, `p(add,1)` is evaluated to callable, and that is correct. In 2nd case, `p(p(call,lambda x,y: x+y),1)` returns a partial, not a fully callable function, so pickle tries to transport it as object containing a function as property. It fails. — Kris, Dec 24 '21 at 10:48

score 0 · Answer 1 · answered Dec 24 '21 at 10:52

A lambda function is an unnamed function. multiprocessing operates by pickle, and that serializes functions by storing their name, e.g. builtins.len:

>>> import pickle
>>> pickle.dumps(len)
b'\x80\x04\x95\x14\x00\x00\x00\x00\x00\x00\x00\x8c\x08builtins\x94\x8c\x03len\x94\x93\x94.'

So when pickling a function built by lambda, the name lookup fails:

>>> pickle.dumps(lambda x: len(x))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7fda31df4280>: attribute lookup <lambda> on __main__ failed

So the two functions have the same call behaviour, but not the same characteristics.

I've tried to make a lambda like a named function by use partial, then fail. I wans to define a named function by a lambda, not by `def`. Use some function in default library in python2 is better for me ... How can? — Hm Renga Y, Dec 29 '21 at 06:51

What's different between these 2 partial function objects?

Precondition

Doubt

1 Answers1