Python: Multiprocessing Does Not Complete Jobs

Question

I'm using python 2.7 with multiprocessing::Pool to run a job in parallel

I've simplified the example below, but here's the main gist of it.

It will create a file for each person in my dict using the apply_async() function. However when I check to see if the file was created properly, I notice that sometimes the file was not created.

Now I'm thinking I've done something wrong in how I used multiprocessing::Pool

Any advice?

import os
from multiprocessing import Pool

def outputFile(person):
    ofh=open(person+'.txt','w')
    ofh.write('test\n')
    ofh.close()

pool = Pool(processes=4)
for person in person_dict:
    pool.apply_async(outputFile,args(person))
pool.close()
pool.join()
for person in person_dict:
    print os.path.isfile(person+'.txt')

True
True
False
True

score 3 · Answer 1 · answered Jun 06 '17 at 20:31

If you do not catch exceptions in sub-processes and print them yourself, you will not see them. The following program produces no output:

import os
from multiprocessing import Pool

def outputFile(person):
    raise Exception("An exception")

pool = Pool(processes=4)
for person in range(100):
    pool.apply_async(outputFile, args=(person,))
pool.close()
pool.join()

You need to catch all exceptions and manually print the traceback:

import os
from multiprocessing import Pool, Lock
import traceback

print_lock = Lock()

def outputFile(person):
    try:
        raise Exception("An exception")
    except:
        with print_lock:
            print "%s: An exception occurred" % person
            print traceback.format_exc()

pool = Pool(processes=4)
for person in range(100):
    args = (person, print_lock)
    pool.apply_async(outputFile, args=(person,))
pool.close()
pool.join()

Output

0: An exception occurred
Traceback (most recent call last):
  File "person.py", line 9, in outputFile
    raise Exception("An exception")
Exception: An exception

1: An exception occurred
Traceback (most recent call last):
  File "person.py", line 9, in outputFile
    raise Exception("An exception")
Exception: An exception

...

99: An exception occurred
Traceback (most recent call last):
  File "person.py", line 9, in outputFile
    raise Exception("An exception")
Exception: An exception

Note: The print_lock is used to keep the output from being interleaved.

score 1 · Answer 2 · answered Jun 06 '17 at 18:16

1

Could this be related to contents of the person_dict?

I have modified your code and run it several times. They all produced the expected results.

Here is the code I modified and tested:

import os
from multiprocessing import Pool

def outputfile(person):
    with open(person+'.txt','w') as ofh:
        ofh.write('test\n')

person_dict = {'a': 'a', 'b': 'b', 'c':'c', 'd':'d'}

pool = Pool(processes=4)
for person in person_dict:
    pool.apply_async(outputfile, (person))
pool.close()
pool.join()

for person in person_dict:
    print(os.path.isfile(person+'.txt'))

answered Jun 06 '17 at 18:16

neurite

2,798
20
32

I'm not sure. Typically I would say 99% of the time I don't get an error. But occasionally the process will drop and an output file is not formed. – DolphinGenomePyramids Jun 06 '17 at 18:26
It could be IO exception or some runtime errors on the child processes that do not show up on the calling process. – neurite Jun 06 '17 at 19:35

Python: Multiprocessing Does Not Complete Jobs

2 Answers2