Executing subprocesses in a Python Application after SIGINT

Question

The following code was taken as a reference from Saving work after a SIGINT

class Main(object):
    def do_stuff(self):
        ...
    def save_work(self):
        ...
    def __init__(self):
        try:
            self.do_stuff()
        except KeyboardInterrupt:
            pass # Or print helpful info
        self.save_work()

This works perfectly fine in case without subprocesses.

However, as soon as you would call a sub process in save_work(), the subprocess will not be executed as it receives the SIGINT signal.

So, executing

    cmd = r"hadoop fs -put '{}' '{}'".format(
        src, dest)
    process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

would not work.

What is a workaround for that?

FYI, that's a very dangerous command. Much safer to take out the `shell=True` and make it `subprocess.Popen(['hadoop', 'fs', '-put', src, dest], stdout=subprocess.PIPE, stderr=subprocess.PIPE)` — Charles Duffy, Jun 18 '18 at 22:20
Consider what happens if someone asks you to upload a file created with `touch $'$(rm -rf ~)\'$(rm -rf ~)\''`, or otherwise containing literal backticks. — Charles Duffy, Jun 18 '18 at 22:21
If you **really** need to use `shell=True`, then take out the literal quotes, and instead use `pipes.quote()` (in Python 2) or `shlex.quote()` (in Python 3) to generate a POSIX-compliant escaping of your names. `cmd = "hadoop fs put {} {}".format(pipes.quote(src), pipes.quote(dest))` still has performance overhead from starting an unnecessary shell, and is prone to side effects due to environment variable-related interference, but it's much less risky than what you have now. — Charles Duffy, Jun 18 '18 at 22:22

Ondrej K. · Answer 1 · 2018-06-18T22:32:28.913

The shortest answer to your problem/question as stated: Replace subprocess.Popen with subprocess.call or one of its (for instance checking) variants. Or add process.communicate().

What happened and why did it look like it "would not work". Popen opened pipes for communication and forked a process as desired. However, the pipe does not have anything reading from it on the parent process side (the one you called it from) which can actually cause the child process (write to stdout/stderr) very quickly land in a blocking I/O. Meanwhile your parent continues to run as there is nothing telling it to wait for its child and eventually terminates at which point the child process receives SIGPIPE (to which default action is to terminate).

Let's have a test.sh:

#!/bin/bash
handle_sigpipe() {
        echo "GOT SIGPIPE" >> OUT
        exit 0
}
trap handle_sigpipe SIGPIPE
echo line1 > OUT
echo line2
echo line3 >> OUT

and a small python script calling it similar to what is in your question:

import time
import subprocess
try:
    time.sleep(20)
except KeyboardInterrupt:
    cmd = "./test.sh"
    process = subprocess.Popen(cmd, shell=True,
                               stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

Now we call and interrupt it:

$ python3 test.py 
^C
$ cat OUT 
line1
GOT SIGPIPE

We've wrote line1, but when script tries to write line2, it just ends up waiting for someone to read from the receiving and of the pipe (at least when called through shell, these stdout writes are line buffered). In the meanwhile, parent finishes up and closes its end of the pipe. test.sh receives SIGPIPE, handler writes it to the file and the shell script terminates.

If you actually actually generally want to perform clean-up/save your work on script exit (also when interrupted). atexit is the common way to do that. If you wanted to generally handle a specific signal (like SIGINT), you could also have a look at signal.signal.

As an aside, `function foo() {` is an amalgam of two different formats -- legacy ksh function syntax, which is just `function foo {` with no `()`, and POSIX sh syntax, which is just `foo() {` with no `function`. Consider using one or the other, to be at least compatible with *some* shell other than bash; see also http://wiki.bash-hackers.org/scripting/obsolete — Charles Duffy, Jun 18 '18 at 22:25

score 0 · Accepted Answer · answered Jul 04 '18 at 17:44

The above stated problem was caused by solved by refactoring the code. Cascading exceptions blocked the signals.

The final solution was.

def signal_handler(sign, frame):
    logger.warn('app has been terminated manually with signal {} at frame {}'.format(sign, frame))
    sys.exit(1)

def end_programm():
    upload_log_file()


def main():
    [...]
    signal.signal(signal.SIGINT, signal_handler)
    signal.signal(signal.SIGTERM, signal_handler)
    atexit.register(end_programm)
    [...]

Executing subprocesses in a Python Application after SIGINT

2 Answers2