Multiprocessing using Pool class in Python giving Pickling error

Question

I am trying to run a simple multiprocessing example in python3.6 in a zeppelin notebook(in windows) but I am not able to execute it. Below is the code that i used:


def sqrt(x):
    return x**0.5

numbers = [i for i in range(1000000)]
with Pool() as pool:
    sqrt_ls = pool.map(sqrt, numbers)

After running this code I am getting the following error:

Traceback (most recent call last):
  File "/tmp/zeppelin_python-3196160128578820301.py", line 315, in <module>
    exec(code, _zcUserQueryNameSpace)
  File "<stdin>", line 6, in <module>
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
    put(task)
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib64/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function sqrt at 0x7f6f84f1a620>: attribute lookup sqrt on __main__ failed

I am not sure if its just me who is facing the issue. As i have seen so many articles where people can run the code easily. If you know the solution please help

Thanks

The only part that is suspicious are the first two steps of the traceback. How do you run this code? — Klaus D., Jan 31 '20 at 06:36
You should add that information prominently to your question. — Klaus D., Jan 31 '20 at 06:41

score 0 · Answer 1 · answered Jan 31 '20 at 15:44

From the multiprocessing documentation:

Note: Functionality within this package requires that the main module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the Pool examples will not work in the interactive interpreter.

Notebooks are running Python interactive interpreters behind the scene, so it's probably why you get this error. You can try to run your code from within a if __name__ == '__main__': statement.

score 0 · Answer 2 · answered Jan 31 '20 at 16:11

A Zeppelin notebook does not emulate a normal module well enough to support the pickling that is used to identify the correct operation to another process. You can put all the functions you want to call into a proper module that you import in the usual fashion.

Multiprocessing using Pool class in Python giving Pickling error

2 Answers2