Unable to catch KeyboardInterrupt exception after starting dask.distributed Client/LocalClient

Question

I'm trying to use Ctrl+C to gracefully stop my running code, including a local dask.distrubted Client. The code below is an example of my setup. When I use Ctrl+C, the stop() method is called properly, however dask Client seems to be improperly exiting/printing a traceback prior to even reaching the self.dask.close() method.

from dask.distributed import Client

class SomeService(object):
    def __init__(self):
        self.dask = None

    def run(self):
        # Setup local dask client
        self.dask = Client()

        while True:
            # Do something blocking

    def stop(self):
        # Close dask client
        print('Closing Dask Client...')
        self.dask.close()
        print('Dask Cient closed.')
        # Stop other stuff
        # ...
        print('SomeService has been stopped.')


if __name__ == '__main__':
    service = SomeService()
    try:
        service.run()
    except KeyboardInterrupt:
        print('Keyboard interrupt received.')
        service.stop()

This is the output I receive:

^CTraceback (most recent call last):
File "<string>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/forkserver.py", line 170, in main
rfds = [key.fileobj for (key, events) in selector.select()]
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/selectors.py", line 560, in select
kev_list = self._kqueue.control(None, max_ev, timeout)
KeyboardInterrupt
Keyboard interrupt received.
Closing Dask Client...
Dask Client closed.
SomeService has been stopped.

I've tried my best to google/stack overflow this issue and the only solution I've found is suggestions to use the signals package to force a callback to SIGINT. This solution however does not work if the SomeService class is to be run in a separate thread because you can only force the signals callback for signals on the main thread.

Any suggestions would be appreciated. Is this not the proper way to hold onto and manage a Dask Client/LocalClient?

Extra Information:

Python 3.5.1
dask==0.19.2
distributed==1.23.2
tornado==5.0.2

score 0 · Answer 1 · answered Oct 29 '18 at 11:57

Use a signal handler that sets a flag. When it is delivered, it causes an IOError exception (with .errno member set to errno.EINTR):

import signal

done = False
def done_handler(signum, frame):
    done = True

if __name__ == '__main__':
    signal.signal(signal.SIGINT, done_handler)

    service = SomeService()

    while not done:
        try:
            service.run()
        except IOError:
            break

    service.stop()

In Python, signals are delivered and handlers executed always in the main thread, so unless caught, the IOError exception should percolate up to the service.run(). (In your example, the KeyboardInterrupt does.) When that exception is caught, the service is stopped.

Unable to catch KeyboardInterrupt exception after starting dask.distributed Client/LocalClient

1 Answers1