Python Process pool executor shutdown on signal

Question

I use supervisor to run some script like this. so when supervisor is stopped or interrupt, i try to gracefully exit from the script. This is my current code

import concurrent.futures
import random
import os
import signal
import sys

executor = concurrent.futures.ProcessPoolExecutor(max_workers=2)

some_flag = True

def some_func():
    while some_flag:
        executor.submit(print_some_num)

def print_some_num():
    print("From worker :{}".format(os.getpid()))
    print(random.randint(0,10))

def handler(arg1, arg2):
    print("Got interrupt")
    some_flag = False
    sys.exit("Run for ur life")
    #executor.shutdown(wait = False) ----> not working
    print("Shutdown")

signal.signal(signal.SIGTERM,handler)
signal.signal(signal.SIGINT,handler)

if __name__ == '__main__':
    some_func()

This works fine and now i am confused when i read about executor.shutdown(wait=True/False). So i tried that and i couldn't get the executor to shutdown (It just hangs). please help me out with these questions

 1) what does executor.shutdown do that sys.exit() doesn't do.
 2) What is advisable in production environment? If executor shutdown is required, please help me fix the issue with shutdown.

score 1 · Accepted Answer · answered May 24 '17 at 16:23

1) Apparently, shutdown stops all the existing messages being processed, but sys.exit() just comes out. (in the docs https://docs.python.org/3/library/concurrent.futures.html)

2) If somebody got stuck in the same place, use some global variable to signal shutdown.

def some_func():
with concurrent.futures.ProcessPoolExecutor(max_workers=2) as executor:
    while True:
        global some_flag
        print(some_flag)
        if not some_flag:
            executor.shutdown(wait=True)
            sys.exit()
        else:
            executor.submit(print_some_num)

def print_some_num():
    print("From worker :{}".format(os.getpid()))
    print(random.randint(0,10))

def handler(arg1, arg2):
    global some_flag
    print("Got interrupt")
    some_flag = False
    print("Shutdown")

signal.signal(signal.SIGTERM,handler)
signal.signal(signal.SIGINT,handler)

Python Process pool executor shutdown on signal

1 Answers1