0

I have a program using python multiprocessing. I find that all the process created in the main programs can be finished but the main programs is always waiting for the return values and can not stop. Can anybody give me some suggestions on how to solve this problem?

The code snippets is as follows:

main program:

workers = multiprocessing.Pool(4)
args = [arg1, arg2, arg3, arg4]
results = workers.map(subfunc, args)
print "we are in main functions "

subfunc(*arg)

# doing some other jobs
result = {.....} # a large dictionary
print 'done with sub functions'
return result # if I change it to  "return 1", it can finish successfully

I can see the output "done with sub functions" for all the processes created in main, but no outputs "we are in main functions". Can anybody help me figure out the problem?

hanqiang
  • 567
  • 1
  • 7
  • 17
  • 2
    Can you test to conform that results has a length? Mayhap this behavior is due to never executing the loop. – mklauber Apr 06 '12 at 19:57
  • It could be that the result dictionary has recursive references, and making it into a pickle doesn't complete. Try writing it out to stdout as a pickle or json. – j13r Apr 06 '12 at 21:38
  • That's what I did right now and it works. Thank you – hanqiang Apr 18 '12 at 19:14

1 Answers1

2

After subfunc returns its result, the child process still needs to pickle up the result and send it through inter-process communication to the master process. So they're printing "done" before the subprocess is actually ready to terminate.

You could try just returning "fake" to make sure that's the problem, and then think about ways to communicate the results between the processes more quickly. For example, depending on the contents of your list, you might be able to package the contents up in a tighter format, or maybe shared memory is a viable option.

Danica
  • 28,423
  • 6
  • 90
  • 122
  • If I return "1" in subfunc, the main program can terminate successfully. So right now I think this problem might be caused by the size of the returned value. In my program, the returned value of subfunc can be as large as 1 GB. How do you think about it? – hanqiang Apr 06 '12 at 21:36
  • @hanqiang Yeah, that's probably it then. What kind of data is it? You probably want to communicate it from the worker to the master in some other way, maybe through shared memory, a [memory-mapped file](http://docs.python.org/library/mmap.html), a database, plain old file i/o, something. – Danica Apr 06 '12 at 21:53
  • Wow... I spent hours trying to figure out why my child processes were telling me they were done, but things were seemingly just freezing. It turned out this pickling was happening and I simply needed to wait. – CodeGuyRoss Feb 11 '17 at 00:13