Using multiprocessing pool in Python

Question

Can someone point out what is wrong with this code snippet. It does not give any results .

    import multiprocessing

results = []
def log_results(result):
    results.append(result)


def multiply(x, y):
    print(f"Gets here for process name {multiprocessing.current_process().name()}")
    return x * y

if __name__ == "__main__":
    pool = multiprocessing.Pool()
    numbers = [(1,1), (2,2), (3,3)]
    for x, y in numbers:
        print (f"Checking x {x} and y {y}")
        pool.apply_async(multiply, (x, y), callback=log_results)
    pool.close()
    pool.join()
    print(results)

results is an empty list which should not be in this case right ? I have used apply_async and map_async . Both do not give the right output. Can someone please help me here

Possible duplicate of [how does the callback function work in python multiprocessing map\_async](https://stackoverflow.com/questions/19699165/how-does-the-callback-function-work-in-python-multiprocessing-map-async) — Mathieu, Apr 05 '19 at 13:01
Use [`starmap_async`](https://docs.python.org/3/library/multiprocessing.html?highlight=process#multiprocessing.pool.Pool.starmap_async) — Mad Physicist, Apr 05 '19 at 13:06
I do not think the function multiply is called in the first place — Subhayan Bhattacharya, Apr 05 '19 at 13:07
Your code works fine when I run it (Python 3.5). It returns [1,4,9] — Hannu, Apr 05 '19 at 13:13

Charles Landau · Accepted Answer · 2019-04-05T16:53:03.723

Edit: You made an edit to your code so now my answer below is out of date. The only two things I think need doing are:

add an error_callback because I still think you need to ensure that the pool as written does not fail silently by default.
rewrite multiprocessing.current_process().name() as multiprocessing.current_process().name.

So:

import multiprocessing

results = []
def log_results(result):
    results.append(result)

def log_e(e):
  print(e)

def multiply(x, y):
    print(f"Gets here for process name {multiprocessing.current_process().name}")
    return x * y


pool = multiprocessing.Pool()
numbers = [(1,1), (2,2), (3,3)]
for x, y in numbers:
    print (f"Checking x {x} and y {y}")
    pool.apply_async(multiply, (x, y), callback=log_results,
                     error_callback=log_e)
pool.close()
pool.join()
print(results)

Old answer

This drove me crazy for a moment but then it made sense.

If I run it with multiply changed like this:

def multiply(nums):
    print("print")
    return nums[0] * nums[1]

It runs fine. You said in the comments "I do not think the function multiply is called in the first place." This is because there is a callback specified but no error_callback specified. The result of omitting an error callback is that your script is failing silently.

You could check this with:

import multiprocessing

results = []
def log_results(result):
    print(result)

def log_e(e):
  print(e)

def multiply(x, y):
    print(f"Gets here for process name {multiprocessing.current_process().name()}")
    return x * y

pool = multiprocessing.Pool()
numbers = [[1,1], [2,2], [3,3]]
mapResult = pool.map_async(multiply, numbers, callback=log_results,
                           error_callback=log_e)

pool.close()
pool.join()

Which gives:

multiply() missing 1 required positional argument: 'y'

And with multiply like so:

def multiply(nums):
    return nums[0] * nums[1]

It then returns [1, 4, 9]

PS I am running Python 3.6.7

d_kennetz · Answer 2 · 2019-04-05T16:35:57.313

So your current code is actually failing because of this line:

 print(f"Gets here for process name {multiprocessing.current_process().name()}")

it errors out as TypeError: 'str' object is not callable, not because there is anything with the way you are calling multiply()

if you remove it:

import multiprocessing

results = []
def log_results(result):
    results.append(result)


def multiply(x, y):
#    print(f"Gets here for process name {multiprocessing.current_process().name()}")
    return x * y

if __name__ == "__main__":
    pool = multiprocessing.Pool()
    numbers = [(1,1), (2,2), (3,3)]
    for x, y in numbers:
        print (f"Checking x {x} and y {y}")
        pool.apply_async(multiply, (x, y), callback=log_results)
    pool.close()
    pool.join()
    print(results)

It returns:

Checking x 1 and y 1
Checking x 2 and y 2
Checking x 3 and y 3
[1, 4, 9]

So if you actually isolate your print(f):

print(multiprocessing.current_process().name())

you get the error: TypeError: 'str' object is not callable because

multiprocessing.current_process()

is actually a process object with name as an attribute of the object which returns a string (thanks darkonaut) string. You are trying to call .name() as a function, but it is an attribute.

So if you change your function to include .name, instead of .name():

import multiprocessing

results = []
def log_results(result):
    results.append(result)


def multiply(x, y):
    print(f"Gets here for process name {multiprocessing.current_process().name}")
    return x * y

if __name__ == "__main__":
    pool = multiprocessing.Pool()
    numbers = [(1,1), (2,2), (3,3)]
    for x, y in numbers:
        print (f"Checking x {x} and y {y}")
        pool.apply_async(multiply, (x, y), callback=log_results)
    pool.close()
    pool.join()
    print(results)

You return:

Checking x 1 and y 1
Checking x 2 and y 2
Checking x 3 and y 3
Gets here for process name ForkPoolWorker-1
Gets here for process name ForkPoolWorker-2
Gets here for process name ForkPoolWorker-3
[1, 4, 9]

Which is what you desire.

Using multiprocessing pool in Python

2 Answers2