How do I add an errback to deferLater?

Question

Consider the following twisted code, using deferLater:

import random

from twisted.internet.task import deferLater
from twisted.internet import reactor

def random_exception(msg='general'):
    if random.random() < 0.5:
        raise Exception("Random exception with 50%% likelihood occurred in %s!" % msg)

def dolater():
    random_exception('dolater')
    print "it's later!"

def whoops(failure):
    failure.trap(Exception)
    print failure

defer = deferLater(reactor, 10, dolater)
defer.addErrback(whoops)

reactor.run()

An exception is raised during the 10 second sleep (namely a KeyboardInterrupt), however, it seems that the whoops method is never called. My assumption is that since I add the errBack after the deferred kicks off, it's never properly registered. Advice appreciated.

EDIT:

Alright, no one likes my use of the signal (not the exception) KeyboardInterrupt to show an error condition outside of the defer. I thought pretty hard about an actual exception that might occur out of the defer callback, but couldn't think of a particularly good one, most everything would be some kind of signal (or developer error), so signal handling is fine for now- but that wasn't really the heart of the question.

As I understand it, twisted's callback/errback system handles errors within the callback structure - e.g. if dolater raises an Exception of some kind. To show this, I have added an exception that could occur during dolater, to show that if the exception occurs in dolater, the errback handles the exception just fine.

My concern was if something went wrong while the reactor was just reacting normally, and the only thing I could get to go wrong was a keyboard interrupt, then I wanted whoops to fire. It appears that if I put other async events into the reactor and raise exceptions from there, then the dolater code wouldn't be affected, and I would have to add errbacks to those other async events. There is no master error handling for an entire twisted program.

So signals it is, until I can find some way to cause the reactor to fail without a signal.

Answer 1

The problem is with the actual exception you're trying to catch, specifically KeyboardInterrupt is not a subclass of Exception, thus can not be catched with it. If you'd just change the line:

failure.trap(Exception)

into:

failure.trap(KeyboardInterrupt)

it surely would catch it. More on Python's exception hierarchy can be found in the official Python docs: https://docs.python.org/2/library/exceptions.html

Answer 2

Twisted is a library for doing many things concurrently. The things are kept as isolated as possible (given that this is still Python, there's still global state, etc).

If you have a TCP server with two clients connect to it and one of them sends you some bad data that triggers a bug in your parser that leads to an exception being raised, that exception isn't going to cause the other client to receive any error. Nor would you want it to, I hope (at least not automatically).

Similarly, if you have a client connected to your server and you start a delayed call with deferLater and the client triggers that bug, you wouldn't want the error to be delivered to the errback on the Deferred returned by deferLater.

The idea here is that separate event sources are generally treated separately (until you write some code that glues them together somehow).

For the ten seconds that are passing between when you call deferLater and when Twisted begins to run the function you passed to deferLater, any errors that happen - including you hitting C-c on your keyboard to make Python raise a KeyboardInterrupt - aren't associated with that delayed call and they won't be delivered to the errback you attach to its Deferred.

Only exceptions raised by your dolater function will cause the errback chain of that Deferred to begin execution.

Answer 3

If by KeyboardInterrupt you mean a signal (ctrl-c, SIGINT, etc), then what you need to do is setup a signal handler with your whoops function as the callback.

By following two previous answers from @jean-paul-calderone twisted: catch keyboardinterrupt and shutdown properly and twisted - interrupt callback via KeyboardInterrupt, I tried the following, and I think it matches your need:

def dolater():
    print "it's later!"

def whoops(signal, stackframe):
    print "I'm here because of signal number " + str(signal)
    reactor.stop()

defer = task.deferLater(reactor, 10, dolater)
signal.signal(signal.SIGINT, whoops)
reactor.run()

That will call whoops on a SIGINT. I put a reactor.stop() in the whoops because otherwise the reactor would just keep on running, take that out if you really want it to keep running in the face of a ctrl-c.

Note: I'm not explicitly showing how to fire a err-back in the signal system because (at least to my understanding) that doesn't really map to how defer'ed should be used. I imagine if you found a way to get the defer'ed into the signal handler you could fire its errback but I think thats out of the expected use-case for twisted and may have crazy consequences.