multiprocessing question using pool and map

Question

I want to get the outputs by using pool and map function in python like this:

**Output in random order:

PoolWorker-10 output: [0, 4, 8, 12, 16, 20]

PoolWorker-11 output: [0, 4, 8, 12, 16, 20]

PoolWorker-12 output: [0, 4, 8, 12, 16, 20]

PoolWorker-1 output: [0, 4, 8, 12, 16, 20]

.....**

I tried several times to write the output result and current_process name together in line but I failed it. I don't know which should be changed in order to get the printed output like that above.

from multiprocessing import Pool, current_process

def multiprocessing_func():
   input_list = [0, 1, 2, 3, 4, 5]
   print("input list: {}".format(input_list))
   print("Output in random order: ")

   pool = Pool(10, current_process_func)
   result = pool.map(calculate, input_list)
   print(result)
   pool.close()
   pool.join()

def calculate(y_list):
   return y_list * 4

def current_process_func():
   print("{} output: ".format(current_process().name))

if __name__ == '__main__':
   multiprocessing_func()

When I run it, I get the result like this below:

Output in random order:

PoolWorker-1 output:

PoolWorker-2 output:

PoolWorker-3 output:

PoolWorker-4 output:

PoolWorker-5 output:

PoolWorker-7 output:

PoolWorker-6 output:

[0, 4, 8, 12, 16, 20]

PoolWorker-8 output:

PoolWorker-9 output:

PoolWorker-10 output:

What should I change to get the result that I want above?

score 0 · Answer 1 · answered Apr 28 '20 at 16:06

Changed your snippet as to be seen below. Overall consider the following:

pool.map(func, data) data should be understood as data to be distributed across the cpu-pool (and is also exclusive to this process). Further at least as many items as processes should be there to make best use of the pool. (I just put 10 identical lists there).

skipped the initialization argument in the Pool() construction to just output the wanted data once the calculation is completed.

from multiprocessing import Pool, current_process

def multiprocessing_func():
    input_list = [0, 1, 2, 3, 4, 5]
    print("input list: {}".format(input_list))
    print("Output in random order: ")

    pool = Pool(10) 
    result = pool.map(calculate, [input_list]*10)

    print()
    print("collected:" , result)
    pool.close()
    pool.join()

def calculate(y_list):
    out = [x*4 for x in y_list] 
    current_process_func(out)
    return out   

def current_process_func(data):
    print("{} output: {}".format(current_process().name, data))

if __name__ == '__main__':
    multiprocessing_func()

leads to:

input list: [0, 1, 2, 3, 4, 5]
Output in random order: 
ForkPoolWorker-52 output: [0, 4, 8, 12, 16, 20]
ForkPoolWorker-55 output: [0, 4, 8, 12, 16, 20]
ForkPoolWorker-57 output: [0, 4, 8, 12, 16, 20]
ForkPoolWorker-51 output: [0, 4, 8, 12, 16, 20]
ForkPoolWorker-53 output: [0, 4, 8, 12, 16, 20]
ForkPoolWorker-54 output: [0, 4, 8, 12, 16, 20]
ForkPoolWorker-56 output: [0, 4, 8, 12, 16, 20]
ForkPoolWorker-58 output: [0, 4, 8, 12, 16, 20]
ForkPoolWorker-60 output: [0, 4, 8, 12, 16, 20]
ForkPoolWorker-59 output: [0, 4, 8, 12, 16, 20]

collected: [[0, 4, 8, 12, 16, 20], [0, 4, 8, 12, 16, 20], [0, 4,    8, 12, 16, 20], [0, 4, 8, 12, 16, 20], [0, 4, 8, 12, 16, 20], [0, 4, 8, 12, 16, 20], [0, 4, 8, 12, 16, 20], [0, 4, 8, 12, 16, 20], [0, 4, 8, 12, 16, 20], [0, 4, 8, 12, 16, 20]]

is this what you expected?

multiprocessing question using pool and map

1 Answers1