Safe way to terminate a python subprocess?

Question

I'm using python's multiprocessing module to handle a number of functions concurrently. Each spawned-process' function gets some initial input arguments, and a Pipe connection to send its results back. For various reasons, I must use individual processes like this, i.e. tools like Pool.map_async()-methods are off the table.

Occasionally, I need to terminate a process which takes too long to finish.

According to the Process documentation:

Warning: If this method is used when the associated process is using a pipe or queue then the pipe or queue is liable to become corrupted and may become unusable by other process. Similarly, if the process has acquired a lock or semaphore etc. then terminating it is liable to cause other processes to deadlock.

I'm not worried about the first part, as each process gets their own pipe object, but how do I determine if a process has 'acquired a lock or semaphore', and/or terminate in a way that's safe for the remainder of my program?

estherwn · Answer 1 · 2020-05-14T09:55:37.587

0

As a side note: It might be worthwhile to check why your subprocesses are taking 'too long to finish'.

As for the warning, it relates to when you 'lock' resources for use. For example:

# function to withdraw from account 
def withdraw(balance, lock):     
    for _ in range(10000): 
        lock.acquire() 
        balance.value = balance.value - 1
        lock.release()

source: https://www.geeksforgeeks.org/synchronization-pooling-processes-python/

If you would terminate the subprocess after it has performed lock.acquire()and before it performed lock.release(), you would have a deadlock situation.

So the question is, do you use any threading.Lock or threading.Semaphore objects in the processes that you want to terminate?

I hope this helps in understanding whether it is safe to terminate your subprocess/thread.

EDIT: By the way, you should also consider using kill() instead of terminate().

edited May 14 '20 at 09:55

answered May 14 '20 at 09:53

estherwn

156
6

Thanks. I don't explicitly do any threading stuff in the process, but I rely on pandas for computations, so I'm worried that that does something like that under the hood. – bjarkemoensted May 14 '20 at 09:55
How does your strategy solve the problem of lost control "inside" the subprocess, to actually do what you propose ( when it got already deadlocked itself)? – user3666197 May 14 '20 at 09:56
AFAIR, the o/s kill will enforce the subprocess to end, yet it is external to the python ecosystem, so the pipes and python Queue-instances will not recognise that from the __main__-side and crash the whole circus even after the o/s kill happened to clear the o/s view of the scene, not the python's one, will they? – user3666197 May 14 '20 at 09:58
1

@ahura, If you do not share your data among threads, pandas will have no reason to lock anything. It could help to share some of your code in the OP if you want us to dive a bit deeper. – estherwn May 14 '20 at 10:10
@user3666197, I am not sure what you mean. All I tried to explain is how locking/deadlocking works, I provide no strategy. About the 'kill' instead of 'terminate': depending on the process, it could handle a 'SIGINT' more neatly. But again, I would need more insight in the application itself. – estherwn May 14 '20 at 10:14

jerry · Answer 2 · 2020-05-14T10:17:16.657

On *nix, you can try sending SIGINT to the process instead of terminating/killing it and catch KeyboardInterrupt exception for the cleanup:

from multiprocessing import Process
import os
import time
import signal

def f():
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        print('Do the emergency cleanup here')

p = Process(target=f)
p.start()
os.kill(p.pid, signal.SIGINT)
p.join()

score -1 · Answer 3 · answered May 14 '20 at 09:54

Q : "Safe way to terminate a python subprocess?"

Well, if there were some, you would never run into this.

If your actual needs can justify the costs of doing so right and well, the best workaround would be to learn from Masters of reliable processing, designing robust, resilient, self-healing systems - like Ms.Margaret HAMILTON (MIT) pioneered in NASA operated Apollo Moon Landing Programme, designing the AGC ( Apollo Guidance Computer ) so right and so well, it could survive its own deadlocking risks, preventing the Eagle lander from crashing the Moon surface.

Best inspirations come from distributed-computing, available for pythonistas if designing safe and self-healing autonomous components using a robust and independent many-node-to-many-node communications-plane frameworks ZeroMQ or nanomsg.

Safe way to terminate a python subprocess?

3 Answers3