python SIGTERM handler is not invoked in multiprocessing

Question

I replicated the following snippet to reproduce an issue I am facing while dealing with SIGTERM handler:

1. a main process, with a SIGTERM handler.

2. a thread spawned from main process

3. a subprocess spawned from the above thread

The intention is to understand the working of SIGTERM handler. I assume that the SIGTERM handler will be inherited by the spawned thread as well as the process. There is a section where deadlock happens (because of the shared queue not being read). This keeps all processes and threads alive as there is a deadlock.

from multiprocessing import Process, Queue
from threading import Thread
import os
import sys
import signal


def sigtermHandlerNew():
    print "SIGTERM received for process: {}".format(os.getpid())
    sys.exit()


def f(q):
    print "f proc id: {}".format(os.getpid())
    q.put('X' * 1000000)


def proc_starter():
    queue = Queue()
    p = Process(target=f, args=(queue,))
    p.start()
    p.join()  # this deadlocks
    obj = queue.get()


def main():
    signal.signal(signal.SIGTERM, sigtermHandlerNew)
    print "main process id: {}".format(os.getpid())
    t = Thread(target=proc_starter)
    t.start()
    t.join()


main()

After I run this program, I will have 2 processes running. I observe a strange behavior here - when I try to kill any of the processes using SIGTERM($ kill -15 <proc-id>), I see that the function for SIGTERM handler is not invoked and this deadlock remains forever(till I signal SIGKILL)

Can someone help me understanding on why the signal is not being honoured by the process? You can directly run this snippet.

Answer 1

Python 2 is plagued by several issues when it comes to mixing threads and signals and it usually is not a recommended practice. You can read some information in regards in the official documentation.

More in depth, what is affecting you is the fact that in Python 2 several primitives which use basic locks cannot be interrupted by signals issue.

This problem does not affect Python 3 anymore so I'd highly recommend you to switch to a newer version of the interpreter.

If you cannot do otherwise, a workaround is to set timeouts to blocking operations such as Queue.get.