What are differences between map(block) and map_async(non block) in python multiprocessing module?

Question

In the previous discussion of stackoverflow:

python-multiprocessing-map-vs-map-async

As quikst3r say:You'll notice map will execute in order, but map_async doesn't.

Here is my example for map(block) vs map_async(non block).

map-sync.py code.

import multiprocessing
import os
import time
from datetime import datetime

def subprocess(number):
    print('this is the %d th subprocess' % number)
    time.sleep(3)  

def mainprocess():
    print('this is the main process ,process number is : %d' % os.getpid())
    pool = multiprocessing.Pool(3)
    list=range(9)
    pool.map(subprocess,list)
    pool.close()
    pool.join()

if __name__ == '__main__':
    mainprocess()

Sometimes map_sync can not execute in order for map function here.

map-async.py code.

import multiprocessing
import os
import time
from datetime import datetime

def subprocess(number):
    print('this is the %d th subprocess' % number)
    time.sleep(3)  

def mainprocess():
    print('this is the main process ,process number is : %d' % os.getpid())
    pool = multiprocessing.Pool(3)
    list=range(9)
    pool.map_async(subprocess,list)
    pool.close()
    pool.join()

if __name__ == '__main__':
    mainprocess()

Sometimes map_async can execute in order for map_async function here.

For multiprocession ,all process are Preemptive Multitasking,no order for map and map_async.

There is no absolute execution in order for map and map_async, and running time are almost same --9 seconds(3*3=9).

Let's see the block in apply function of multiprocessing module.

apply.py code.

import multiprocessing
import os
import time
from datetime import datetime

def subprocess(number):
    print('this is the %d th subprocess' % number)
    time.sleep(3)  

def mainprocess():
    print('this is the main process ,process number is : %d' % os.getpid())
    pool = multiprocessing.Pool(3)
    for i in range(9):    
        pool.apply(subprocess,args=(i,))
    pool.close()
    pool.join()

if __name__ == '__main__':
    mainprocess()

27=3*9 (all processes blocked)

I am puzzled how to demonstrate the block and non-block attributions between map and map_async in multiprocessing module?
What are the differences between map(block) and map_async(non block) multiprocessing module?

Answer 1

Maybe you misunderstand the quikst3r's words.

pool create many worker process to do something, about the block or not, It means whether the worker process will block the main process or not. And map will block the main process util it finish its work, and map_asyc will not block the main process, so in map_async all processes will be going ahead at the same time. They do not ensure the worker process's order.

 import multiprocessing
import os   
import time 
from datetime import datetime

def subprocess(number):
    print('this is the %d th subprocess' % number)
    time.sleep(3)  

def mainprocess():
    print('this is the main process ,process number is : %d' % os.getpid())
    pool = multiprocessing.Pool(3)
    list=range(9)
    pool.map(subprocess,list)    # or map_aysnc                                                                                                                                                             
    print "hallo"
    print "hallo again"
    pool.close()
    pool.join()

if __name__ == '__main__':
    mainprocess()

if it is map, the output will be like this:

this is the main process ,process number is : 24812
this is the 0 th subprocess
this is the 1 th subprocess
this is the 2 th subprocess
this is the 3 th subprocess
this is the 4 th subprocess
this is the 5 th subprocess
this is the 6 th subprocess
this is the 7 th subprocess
this is the 8 th subprocess
hallo
hallo again

And if it is map_async, the output is:

this is the main process ,process number is : 24775
hallo
hallo again
this is the 0 th subprocess
this is the 1 th subprocess
this is the 2 th subprocess
this is the 3 th subprocess
this is the 4 th subprocess
this is the 5 th subprocess
this is the 6 th subprocess
this is the 7 th subprocess
this is the 8 th subprocess

As for apply or apply_async, It's meaning about block is same as map or map_async. It is saying that the worker process block the main process or not.