concurrent.futures ProcessPoolExecutor restarts script for every Process

Question

I am currently experimenting with ProcessPoolExecutor instead of Threadpools, because i need an alternative that i can cancel when a network request doesn't answer. Now, i stumbled upon a strange side effect of using the processpoolexecutor: For every process, the whole script gets restarted.

Example:

import concurrent.futures
import math

print('TEST')

PRIMES = [
    112272535095293,
    112582705942171,
    112272535095293,
    115280095190773,
    115797848077099,
    1099726899285419]

def is_prime(n):
    if n < 2:
        return False
    if n == 2:
        return True
    if n % 2 == 0:
        return False

    sqrt_n = int(math.floor(math.sqrt(n)))
    for i in range(3, sqrt_n + 1, 2):
        if n % i == 0:
            return False
    return True

def main():
    with concurrent.futures.ProcessPoolExecutor() as executor:
        for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
            print('%d is prime: %s' % (number, prime))

if __name__ == '__main__':
    main()

Output:

TEST
TEST
TEST
TEST
TEST
TEST
TEST
112272535095293 is prime: True
112582705942171 is prime: True
112272535095293 is prime: True
115280095190773 is prime: True
115797848077099 is prime: True
1099726899285419 is prime: False

For every Process, the 'TEST' gets printed again. Why is that? Can i prevent this from happening? I was not yet able to find anything on that topic.

score 0 · Answer 1 · answered Jan 23 '23 at 17:12

If I understand it correctly, each process worker imports current file and then runs the specified function - so that's why the print('TEST') gets printed for each time.

If you want to prevent it, move the print into the main function which is "protected" by if __name__ = '__main__' construction - that means it's run only when it's executed as the "main programm" and not when the file is imported.

See: https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor

The main module must be importable by worker subprocesses. This means that ProcessPoolExecutor will not work in the interactive interpreter.

concurrent.futures ProcessPoolExecutor restarts script for every Process

1 Answers1