Multiprocessing - Print output of process inconsistent

Question

Im having trouble getting consistent logging from within a process in a process pool while using map.

Here is the function being called in the process pool:

from multiprocessing import Pool, Lock
lock = Lock()

def ping(host):
name, address = host

with lock:
    sys.stdout.write("\npinging " + name + " at address " + address)
    sys.stdout.flush()

pop = Popen(['ping', '-n', '1', address], stdout=PIPE)

pop.communicate() # We use communicate to not block the stdin stream, since we are going to need the return code from it
res = pop.wait() # This gets us the return code we want

status = "OK"

if res != 0:
    status = "FAILED"

with lock:
    sys.stdout.write("\n" + name + " , " + address + " , " + status + "\n")
    sys.stdout.flush()

return (name, address, status)

pool declaration:

pool = Pool(processes=4)
results = pool.map(ping, hosts)

print results

For some reason, im getting pretty inconsistent output from this. I assume it is because more than one process tries to print at a given moment, however, shouldnt Lock() solve this?

Here are the types of outputs im getting:

this i'd consider good:

pinging CP_RESERVED_MS at address 12.13.14.15
pinging test at address 1.1.1.1
pinging test_diplicate at address 1.1.1.2


CP_RESERVED_MS , 12.13.14.15 , FAILED
test_diplicate , 1.1.1.2 , FAILED
test , 1.1.1.1 , FAILED
[(u'CP_RESERVED_MS', u'12.13.14.15', 'FAILED'), (u'test', u'1.1.1.1', 'FAILED'), (u'test_diplicate', u'1.1.1.2', 'FAILED')]

Then suddenly it gets jumbled:

pinging CP_RESERVED_MS at address 12.13.14.15pinging test at address 1.1.1.1
pinging test_diplicate at address 1.1.1.2

test_diplicate , 1.1.1.2 , FAILED

CP_RESERVED_MS , 12.13.14.15 , FAILED
test , 1.1.1.1 , FAILED
[(u'CP_RESERVED_MS', u'12.13.14.15', 'FAILED'), (u'test', u'1.1.1.1', 'FAILED'), (u'test_diplicate', u'1.1.1.2', 'FAILED')]

wont new-line, missing new-lines, etc

why is this?

p.s im using windows 10 64 bit

Which lock are you using? multiprocessing.Lock? And where do you create it? — janbrohl, Jul 30 '16 at 18:16
If possible, show a compete program. For example, we can't see here how you created the lock. Also say which OS you're using. For example, there are things that work on Linux-y systems that can't work on Windows, due to fundamental differences in the way `multiprocessing` is implemented across platforms. — Tim Peters, Jul 30 '16 at 18:18
@janbrohl I just use the lock object i get when i import it from the module. it is assigned to a global scoped variable — Daedalus, Jul 30 '16 at 18:56
@TimPeters this is pretty much the complete program, i just removed the if name == main and a couple other stuff to make it cleaner, but nothing of value is omitted — Daedalus, Jul 30 '16 at 18:57
You didn't say which OS. Now that I can see how `lock` was defined, the _way_ you defined it can't work on Windows - but I'm not going to go through the considerable bother of explaining how to fix that case unless I know it's relevant ;-) — Tim Peters, Jul 30 '16 at 19:05
OK! Now that I know it's WIndows, I voted to close because it's essentially a duplicate of a question I answered in detail a few days ago: http://stackoverflow.com/questions/38626817/missing-lines-when-writing-file-with-multiprocessing-lock-python/38627411#38627411 — Tim Peters, Jul 30 '16 at 19:11
@TimPeters Ah! i see, ive implemented what you wrote and it worked! Thanks a bunch! However, i do have a few questions. 1. do i still need the `with lock`? 2. was i right in assuming i cant use `print` here? if so, why? — Daedalus, Jul 30 '16 at 21:24
You do need `with lock:` - a lock accomplishes nothing unless you use it ;-) I don't know why you thought you couldn't use `print`. I use it all the time, but I'm usually using the latest Python 3. There are _possibly_ issues with inheriting (across processes) file handles on Windows under earlier versions. The implementation of this stuff is, alas, inherently complex :-( — Tim Peters, Jul 30 '16 at 21:34

Multiprocessing - Print output of process inconsistent

0 Answers0