21

How do I send a Ctrl-C to multiple ssh -t processes in Popen() objects?

I have some Python code that kicks off a script on a remote host:

# kickoff.py

# i call 'ssh' w/ the '-t' flag so that when i press 'ctrl-c', it get's
# sent to the script on the remote host.  otherwise 'ctrol-c' would just
# kill things on this end, and the script would still be running on the
# remote server
a = subprocess.Popen(['ssh', '-t', 'remote-host', './script.sh', 'a'])
a.communicate()

That works great, but I need to kick off multiple scripts on the remote host:

# kickoff.py

a = subprocess.Popen(['ssh', '-t', 'remote-host', './script.sh', 'a'])
b = subprocess.Popen(['ssh', '-t', 'remote-host', './script.sh', 'b'])
a.communicate()
b.communicate()

The result of this is that Ctrl-C doesn't reliably kill everything, and my terminal always gets garbled afterwards (I have to run 'reset'). So how can I kill both remote scripts when the main one is killed?

Note: I'm trying to avoid logging into the remote-host, searching for 'script.sh' in the process list, and sending a SIGINT to both of the processes. I just want to be able to press Ctrl-C on the kickoff script, and have that kill both remote processes. A less optimal solution may involve deterministically finding the PID's of the remote scripts, but I don't know how to do that in my current set-up.

Update: the script that gets kicked off on the remote server actually starts up several children processes, and while killing the ssh does kill the original remote script (probably b/c of SIGHUP), the children tasks are not killed.

aaronstacy
  • 6,189
  • 13
  • 59
  • 72
  • I changed the title to something that actually describes what you want to do. – Lennart Regebro Jan 12 '11 at 13:34
  • No idea if it will work, but have you tried sending the end of text byte "\x03" to the subprocess? That's equivalent to Ctrl-C. – Thomas K Jan 12 '11 at 16:31
  • @Thomas K: Good thinking, but unfortunately that will only work if the "\x03" is sent to the input side of a terminal the process is attached to (or of course if the program interprets the data that way!)... sadly in this case the subprocess is via a pipe rather than a terminal, so the handling that converts Ctrl-C into SIGINT isn't there :( – psmears Jan 12 '11 at 17:08
  • What remote program garble the terminal ? – BatchyX Jan 12 '11 at 17:25
  • When I say "my terminal gets garbled," I mean that it stops outputting '\r' characters and it won't echo what I type. Once I run 'reset,' it's back to normal. – aaronstacy Jan 12 '11 at 20:01

4 Answers4

11

The only way I was able to successfully kill all of my child processes was by using pexpect:

a = pexpect.spawn(['ssh', 'remote-host', './script.sh', 'a'])
a.expect('something')

b = pexpect.spawn(['ssh', 'remote-host', './script.sh', 'b'])
b.expect('something else')

# ...

# to kill ALL of the children
a.sendcontrol('c')
a.close()

b.sendcontrol('c')
b.close()

This is reliable enough. I believe someone else posted this answer earlier, but then deleted the answer, so I will post it in case someone else is curious.

aaronstacy
  • 6,189
  • 13
  • 59
  • 72
4

When killed, ssh will send a SIGHUP to the remote processes. You could wrap the remote processes into a shell or python script that will kill them when that script receives a SIGHUP (see the trap command for bash, and the signal module in python)

It might even be possible to do it with a bloated command line instead of a remote wrapper script.

The problem is that killing the remote processes is not what you want, what you want is to have a working terminal after you do Ctrl+C. to do that, you will have to kill the remote processes AND see the remaining output, which will contain some terminal control sequences to reset the terminal to a proper state. For that you will need a mecanism to signal a wrapper script to kill the processes. This is not the same thing.

BatchyX
  • 4,986
  • 2
  • 18
  • 17
  • The default action for SIGHUP is to terminate, so you may not even need a wrapper. – psmears Jan 12 '11 at 17:04
  • 2
    @psmears: If that was the case, the asker wouldn't have any problem and the question wouldn't exist. – BatchyX Jan 12 '11 at 17:15
  • I was assuming that that was because the signals weren't being delivered locally correctly for some reason (i.e. an ssh was being left running in the background somehow). – psmears Jan 12 '11 at 18:10
  • This is correct, but it does not seem to work for me, I think because the script I start goes on to start several other scripts, so only the original script that was started dies. – aaronstacy Jan 13 '11 at 04:06
3

I haven't tried this, but maybe you can catch a KeyboardInterrupt and then kill the processes:

try
    a = subprocess.Popen(['ssh', '-t', 'remote-host', './script.sh', 'a'])
    b = subprocess.Popen(['ssh', '-t', 'remote-host', './script.sh', 'b'])
    a.communicate()
    b.communicate()
except KeyboardInterrupt:
    os.kill(a.pid, signal.SIGTERM)
    os.kill(b.pid, signal.SIGTERM)
lauret
  • 31
  • 1
  • 1
    Popen objects in fact have a `.kill()` method. But as singularity says, the challenge is to kill the remote processes, not the local ones. I don't think there's any simple way to do it, because Python only knows about the local processes. – Thomas K Jan 12 '11 at 16:14
  • 2
    @singularity: You're right that that kills the local process, but that will usually also kill the remote processes too as a consequence (see BatchyX's answer). – psmears Jan 12 '11 at 17:05
1

I worked around a similar problem this problem by unmapping all of the signals I cared about. When Ctrl+C is pressed, it will still be passed through to the subprocess but Python will wait until the subprocess exits before handling the signal in the main script. This works fine for a signal subprocess as long as the subprocess responds to Ctrl+C.

class DelayedSignalHandler(object):
    def __init__(self, managed_signals):
        self.managed_signals = managed_signals
        self.managed_signals_queue = list()
        self.old_handlers = dict()

    def _handle_signal(self, caught_signal, frame):
        self.managed_signals_queue.append((caught_signal, frame))

    def __enter__(self):
        for managed_signal in self.managed_signals:
            old_handler = signal.signal(managed_signal, self._handle_signal)
            self.old_handlers[managed_signal] = old_handler

    def __exit__(self, *_):
        for managed_signal, old_handler in self.old_handlers.iteritems():
            signal.signal(managed_signal, old_handler)

        for managed_signal, frame in self.managed_signals_queue:
            self.old_handlers[managed_signal](managed_signal, frame)

Now, my subprocess code looks like this:

    with DelayedSignalHandler((signal.SIGINT, signal.SIGTERM, signal.SIGHUP)):
        exit_value = subprocess.call(command_and_arguments)

Whenever Ctrl+C is pressed, the application is allowed to exit before the signal is handled so you don't have to worry about the terminal getting garbled because the subprocess thread was not terminated at the same time as the main process thread.

Eric Pruitt
  • 1,825
  • 3
  • 21
  • 34