Why do I see extra line breaks when working with python multiprocessing pool?

Question

Example:

from multiprocessing.dummy import Pool as ThreadPool

def testfunc(string):
    print string

def main():

    strings = ['one', 'two', 'three', ...]
    pool = ThreadPool(10)
    results = pool.map(testfunc, strings)
    pool.close()
    pool.join()

if __name__ == '__main__':
    main()

This will not give us clear results with one result in one line:

one
two 
three

But mesh, that has random linebreaks, like

one 


two
three

four
five
...

Why does it happen? Can i output my data with one linebreak per function call?

P.S. Sometimes i have even no linebreaks or even spaces! P.P.S. Working under windows

score 4 · Accepted Answer · answered Oct 01 '14 at 14:01

print is a non-atomic operation, so one print can be interrupted in the middle by another print in a different process. You can prevent two processes from calling print simultaneously by putting a Lock around it.

from multiprocessing.dummy import Pool as ThreadPool
from multiprocessing import Lock

print_lock = Lock()
def testfunc(string):
    print_lock.acquire()
    print string
    print_lock.release()

def main():

    strings = ['one', 'two', 'three', 'four', 'five']
    pool = ThreadPool(10)
    results = pool.map(testfunc, strings)
    pool.close()
    pool.join()

if __name__ == '__main__':
    main()

score 3 · Answer 2 · answered Sep 09 '21 at 10:41

3

Five years too late, but I think it may be worth mentioning that a quick and dirty solution to this is to explicitly include the linebreak at the end of the print:

print('message\n', end='')

answered Sep 09 '21 at 10:41

OlleNordesjo

133
1
7

falsetru · Answer 3 · 2014-10-01T14:17:11.983

2

Because workers (processes/threads according to the pool you're using) are not synchronized. You can use lock.

Or, instead of printing output in worker-processes, you can print it in main process.

def testfunc(string):
    return string

def main():
    strings = ['one', 'two', 'three', ...]
    pool = ThreadPool(10)
    results = pool.map(testfunc, strings)
    for result in results:
        print result
    pool.close()
    pool.join()

edited Oct 01 '14 at 14:17

answered Oct 01 '14 at 14:01

falsetru

357,413
63
732
636

@dano, You're right. `multiprocessing.dummy.Pool` return `multiprocessing.pool.ThreadPool`. Removed the part accordingly. Thank you for pointing that. – falsetru Oct 01 '14 at 14:11

score 1 · Answer 4 · answered Oct 01 '14 at 14:03

All the threads are writing to the same output file, in this case it is stdout. So, before even one process finishes writing, the other threads are also writing to the same output file. Instead, you can gather the results from all the threads and print them at the main itself, like this

def testfunc(string):
    return string

...
...

    print "\n".join(pool.map(testfunc, strings))

Why do I see extra line breaks when working with python multiprocessing pool?

4 Answers4