Twisted Trial: How to test a MultiService with a client in it -- Reactor was unclean

Question

I've inherited a Twisted MultiService that I'm trying to add tests to, but whatever I do, I end up with a DirtyReactorAggregateError. The service connects to a server with a twisted.application.internet.TCPClient. I think the error is because the TCPClient is not disconnecting, but I'm not sure how I'm supposed to disconnect it. What is the proper way to test a Twisted Service with a client in it?

Here's the test case:

from labrad.node import *
from twisted.trial import unittest
import os
from socket import gethostname

class NodeTestCase(unittest.TestCase):

    def setUp(self):
        import labrad
        name = os.environ.get('LABRADNODE', gethostname()) + '_test'
        self.node = Node(name,
            labrad.constants.MANAGER_HOST,
            labrad.constants.MANAGER_PORT)
        self.node.startService()
        #self.addCleanup(self.node.stopService)

    def test_nothing(self):
        self.assertEqual(3, 3)

    def tearDown(self):
        return self.node.stopService()

Here's the Node service itself:

class Node(MultiService):
    """Parent Service that keeps the node running.

    If the manager is stopped or we lose the network connection,
    this service attempts to restart it so that we will come
    back online when the manager is back up.
    """
    reconnectDelay = 10

    def __init__(self, name, host, port):
        MultiService.__init__(self)
        self.name = name
        self.host = host
        self.port = port

    def startService(self):
        MultiService.startService(self)
        self.startConnection()

    def startConnection(self):
        """Attempt to start the node and connect to LabRAD."""
        print 'Connecting to %s:%d...' % (self.host, self.port)
        self.node = NodeServer(self.name, self.host, self.port)
        self.node.onStartup().addErrback(self._error)
        self.node.onShutdown().addCallbacks(self._disconnected, self._error)
        self.cxn = TCPClient(self.host, self.port, self.node)
        self.addService(self.cxn)

    def stopService(self):
        if hasattr(self, 'cxn'):
            d = defer.maybeDeferred(self.cxn.stopService)
            self.removeService(self.cxn)
            del self.cxn
            return defer.gatherResults([MultiService.stopService(self), d])
        else:
            return MultiService.stopService(self)

    def _disconnected(self, data):
        print 'Node disconnected from manager.'
        return self._reconnect()

    def _error(self, failure):
        r = failure.trap(UserError)
        if r == UserError:
            print "UserError found!"
            return None
        print failure.getErrorMessage()
        return self._reconnect()

    def _reconnect(self):
        """Clean up from the last run and reconnect."""
        ## hack: manually clearing the dispatcher...
        dispatcher.connections.clear()
        dispatcher.senders.clear()
        dispatcher._boundMethods.clear()
        ## end hack
        if hasattr(self, 'cxn'):
            self.removeService(self.cxn)
            del self.cxn
        reactor.callLater(self.reconnectDelay, self.startConnection)
        print 'Will try to reconnect in %d seconds...' % self.reconnectDelay

Answer 1

You should refactor your service so that it can use something like MemoryReactor from twisted.test.proto_helpers (the one public module in the twisted.test package, although hopefully it will move out of twisted.test eventually).

The way that you use MemoryReactor is to pass it into your code as the reactor to use. If you then want to see what happens when the connection succeeds, look at some of its public attributes - tcpClients for connectTCP, tcpServers for listenTCP, etc. Your tests can then pull out the Factory instances which were passed to connectTCP/listenTCP, etc, and call buildProtocol on them and makeConnection on the result. To get an ITransport implementation you can use twisted.test.proto_helpers.StringTransportWithDisconnection. You might even look at the (private API! be careful! it'll break without warning! although it really should be public) twisted.test.iosim.IOPump to relay traffic between clients and servers.

If you really actually need to do whole-system non-deterministic real-world testing, with all the complexity and random unrelated failures that implies, here's an article on actually shutting down a client and server all the way.

Answer 2

I had a similar issue with trying to test a instance of an application. I ended up creating a Python base class that used setUp and tearDown methods to start/stop the application.

from twisted.application.app import startApplication
from twisted.application.service import IServiceCollection
from twisted.internet.defer import inlineCallbacks
from twisted.trial import unittest

class MyTest(unittest.TestCase):

    def setUp(self):
        startApplication(self.app, save=False)

    @inlineCallbacks
    def tearDown(self):
        sc = self.app.getComponent(IServiceCollection)
        yield sc.stopService()