Python 2.7: process pool not working (inifinite workers created)

Question

My multiprocess pool is not working and is behaving strangely:

from multiprocessing import Pool
import multiprocessing, logging
logger = multiprocessing.log_to_stderr()
logger.setLevel(multiprocessing.SUBDEBUG)


def f(x):
    return x*x

if __name__ == '__main__':
    pool = Pool(processes=2)
    print pool.map(f,range(10))

The logger prints out a whole bunch of stuff like:

[DEBUG/MainProcess] created semlock with handle 420
[DEBUG/MainProcess] created semlock with handle 432
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] cleaning up worker 1
[DEBUG/MainProcess] cleaning up worker 0
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] cleaning up worker 1
[DEBUG/MainProcess] cleaning up worker 0
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] cleaning up worker 1
[DEBUG/MainProcess] cleaning up worker 0
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] cleaning up worker 1
[DEBUG/MainProcess] cleaning up worker 0
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] cleaning up worker 1
[DEBUG/MainProcess] cleaning up worker 0
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] cleaning up worker 1
[DEBUG/MainProcess] cleaning up worker 0
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] cleaning up worker 1
[DEBUG/MainProcess] cleaning up worker 0
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] cleaning up worker 1
[DEBUG/MainProcess] cleaning up worker 0
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] cleaning up worker 1
[DEBUG/MainProcess] cleaning up worker 0
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] cleaning up worker 1
[DEBUG/MainProcess] cleaning up worker 0
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] added worker

...ad infinitum and never prints the result.

Why isn't this working?

Are you using an IDE? Try running the script from the command line. — unutbu, Mar 19 '13 at 21:33
The code works fine for me. Try running it from command line as suggested by @unutbu. — crayzeewulf, Mar 19 '13 at 21:34

score 3 · Accepted Answer · answered Mar 19 '13 at 21:37

3

The issue was caused by running the code in PyScripter (v2.5.3.0) on windows. Running from the command line worked fine.

answered Mar 19 '13 at 21:37

Muhd

24,305
22
61
78

score 1 · Answer 2 · answered Mar 19 '13 at 21:32

FWIW, your code works fine on my box. (Windows 7, x64, Python 2.7.0):

C:\Users\X\Desktop>python mp.py
[DEBUG/MainProcess] created semlock with handle 384
[DEBUG/MainProcess] created semlock with handle 396
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] added worker
[DEBUG/PoolWorker-2] recreated blocker with handle 12
[DEBUG/PoolWorker-2] recreated blocker with handle 12
[DEBUG/PoolWorker-2] recreated blocker with handle 24
[[DEBUG/PoolWorker-1] recreated blocker with handle 32
DEBUG/PoolWorker-2] recreated blocker with handle 24
[DEBUG/PoolWorker-1] recreated blocker with handle 32
[[DEBUG/PoolWorker-1] recreated blocker with handle 48
INFO/PoolWorker-2] child process calling self.run()
[DEBUG/PoolWorker-1] recreated blocker with handle 48
[INFO/PoolWorker-1] child process calling self.run()
[[INFO/PoolWorker-1] child process calling self.run()
INFO/PoolWorker-2] child process calling self.run()
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[SUBDEBUG/MainProcess] calling <Finalize object, callback=_terminate_pool, args=(<Queue.Queue instance at 0x01FCCBE8>, <multiprocessing.queues.SimpleQueue object at 0x01F57890>, <multiprocessing.queues.SimpleQueue object
 at 0x01FAFBB0>, [<Process(PoolWorker-1, started daemon)>, <Process(PoolWorker-2, started daemon)>], <Thread(Thread-1, started daemon 84304)>, <Thread(Thread-2, started daemon 84900)>, <Thread(Thread-3, started daemon 42
352)>, {}), exitprority=15>
[SUBDEBUG/MainProcess] finalizer calling <bound method type._terminate_pool of <class 'multiprocessing.pool.Pool'>> with args (<Queue.Queue instance at 0x01FCCBE8>, <multiprocessing.queues.SimpleQueue object at 0x01F5789
0>, <multiprocessing.queues.SimpleQueue object at 0x01FAFBB0>, [<Process(PoolWorker-1, started daemon)>, <Process(PoolWorker-2, started daemon)>], <Thread(Thread-1, started daemon 84304)>, <Thread(Thread-2, started daemo
n 84900)>, <Thread(Thread-3, started daemon 42352)>, {}) and kwargs {}
[DEBUG/MainProcess] finalizing pool
[DEBUG/MainProcess] helping task handler/workers to finish
[DEBUG/MainProcess] task handler got sentinel
[DEBUG/MainProcess] removing tasks from inqueue until task handler finished
[DEBUG/MainProcess] task handler sending sentinel to result handler
[DEBUG/MainProcess] result handler got sentinel
[DEBUG/MainProcess] task handler sending sentinel to workers
[[DEBUG/MainProcess] ensuring that outqueue is not full
DEBUG/PoolWorker-1] worker got sentinel -- exiting
[[[DEBUG/PoolWorker-2] worker got sentinel -- exiting
DEBUG/MainProcess] task handler exiting
DEBUG/PoolWorker-1] worker got sentinel -- exiting
[[DEBUG/PoolWorker-1] worker exiting after 5 tasks
[[DEBUG/PoolWorker-1] worker exiting after 5 tasks
DEBUG/PoolWorker-2] worker got sentinel -- exiting
[INFO/PoolWorker-1] process shutting down
DEBUG/MainProcess] terminating workers
[DEBUG/MainProcess] result handler exiting: len(cache)=0, thread._state=2
[DEBUG/MainProcess] worker handler exiting
[DEBUG/MainProcess] joining task handler
[DEBUG/MainProcess] joining result handler
[DEBUG/MainProcess] joining pool workers
[DEBUG/MainProcess] running the remaining "atexit" finalizers
C:\Users\X\Desktop>

Python 2.7: process pool not working (inifinite workers created)

2 Answers2