-1

I am working on multiprocessing and trying to replicate the code given in the below link:

Python Multiprocessing imap

My system is hanging in both Spyder and Jupyter as shown following. What could be the reason?

enter image description here

Following is the code exactly copied and running. But it is just hanging.

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(3) as p:
        print(p.map(f, [1, 2, 3]))
mkrieger1
  • 19,194
  • 5
  • 54
  • 65
Chakra
  • 647
  • 1
  • 8
  • 16
  • 1
    Your code is different from what you link to - specifically you haven’t included the line `if __name__=‘__main__’:` - why not? Try adding it to the code. FYI pretty much every example in the documentation for multiprocessing includes this line; it’s not there by accident see https://docs.python.org/3/library/multiprocessing.html – DisappointedByUnaccountableMod Jun 05 '21 at 09:13
  • HI Barny, It seems my system issue. I am still not sure what is happening. I already went through that link which you pasted. It hanged too much time and it showed following message after an hour: > – Chakra Jun 05 '21 at 09:56
  • Did you add the `if __name—`… line? Did you try running the code from commandline rather than in Jupyter or Spyder? – DisappointedByUnaccountableMod Jun 05 '21 at 10:06
  • This is the code I am running: from multiprocessing import Pool def f(x): return x*x if __name__ == '__main__': with Pool(3) as p: print(p.map(f, [1, 2, 3])) – Chakra Jun 05 '21 at 10:13
  • Edit your code into your question _as text_. Also try running your code from the commandline - does that work? – DisappointedByUnaccountableMod Jun 05 '21 at 10:15
  • 1
    IPython Notebooks have a myraiad of problems with multiprocessing in general.. They can be worked around, but I find it's easier just to use a more traditional IDE which calls your code with a normal python interpreter. – Aaron Jun 05 '21 at 15:30
  • 1
    This is my personal opinion here, but I think Jupyter and all other IPython Notebook programs are only really useful for live presentation where it needs to be somewhat "Pretty". Getting real code written is much better done with python itself (Cpython to be specific) because that's what everything's designed to work with. No messing around with "Oh I need to use this one special trick that's not in the documentation to get something to work". – Aaron Jun 05 '21 at 15:39
  • 1
    If you read the [docs on multiprocessing](https://docs.python.org/3/library/multiprocessing.html) starting with the section beginning with: *Note Functionality within this package requires that the \_\_main__ module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the multiprocessing.pool.Pool examples will not work in the interactive interpreter. For example:, you will see this will not work.* **Put function `f` in another *.py* file and import it to get it to work.** – Booboo Jun 06 '21 at 10:34
  • HI Booboo, I gave up this task but with your comment really surprised me. It actually worked. Thanks a lot sir. Can't understand how to approve this answer. – Chakra Jun 06 '21 at 15:19
  • I took my comment and created a proper answer from it, if you wish to *approve* that. You might also wish to delete your answer, which now is not required. – Booboo Jun 07 '21 at 17:04

1 Answers1

1

If you read the docs on multiprocessing, in particular the following section:

enter image description here

... you will see this will not work. The solution is to put function f in another .py file and import it order to get it to work. For example:

File worker.py:

def f(x):
    return x*x

Your revised code:

from multiprocessing import Pool
from worker import f

if __name__ == '__main__':
    with Pool(3) as p:
        print(p.map(f, [1, 2, 3]))
Booboo
  • 38,656
  • 3
  • 37
  • 60