Twisted - how to make lots of Python code non-blocking

Question

I've been trying to get this script to perform the code in hub() in written order.

hub() contains a mix of standard Python code and requests to carry out I/O using Twisted and Crossbar.

However, because the Python code is blocking, reactor doesn't have any chance to carry out those 'publish' tasks. My frontend receives all the published messages at the end.

This code is a massively simplified version of what I'm actually dealing with. The real script (hub() and the other methods it calls) is over 1500 lines long. Modifying all those functions to make them non-blocking is not ideal. I'd rather be able to isolate the changes to a few methods like publish() if that's possible to fix this problem.
I have played around with terms like async, await, deferLater, loopingCall, and others. I have not found an example that helped yet in my situation.

Is there a way to modify publish() (or hub()) so they send out the messages in order?

from autobahn.twisted.component import Component, run
from twisted.internet.defer import inlineCallbacks, returnValue
from twisted.internet import reactor, defer

component = Component(
    transports=[
        {
            u"type": u"websocket",
            u"url": u"ws://127.0.0.1:8080/ws",
            u"endpoint": {
                u"type": u"tcp",
                u"host": u"localhost",
                u"port": 8080,
            },
            u"options": {
                u"open_handshake_timeout": 100,
            }
        },
    ],
    realm=u"realm1",
)

@component.on_join
@inlineCallbacks
def join(session, details):
    print("joined {}: {}".format(session, details))

    def publish(context='output', value='default'):
        """ Publish a message. """
        print('publish', value)
        session.publish(u'com.myapp.universal_feedback', {"id": context, "value": value})

    def hub(thing):
        """ Main script. """
        do_things
        publish('output', 'some data for you')
        do_more_things
        publish('status', 'a progress message')
        do_even_more_things
        publish('status', 'some more data')
        do_all_the_things
        publish('other', 'something else')

    try:
        yield session.register(hub, u'com.myapp.hello')
        print("procedure registered")
    except Exception as e:
        print("could not register procedure: {0}".format(e))


if __name__ == "__main__":
    run([component])
    reactor.run()

score 0 · Answer 1 · answered Feb 20 '18 at 13:50

Your join() function is async (decorated with @inlineCallbacks and contains at least one yield in the body).

Internally it registers function hub() as WAMP RPC; hub() is however not async.

Also the calls to session.publish() are not yielded as async calls should be.

Result: you add a bunch of events to the eventloop but don't await them until you flush the eventloop on application shutdown.

score 0 · Answer 2 · answered Jul 10 '18 at 12:31

You need to make your function hub and publish async.

@inlineCallbacks
def publish(context='output', value='default'):
    """ Publish a message. """
    print('publish', value)
    yield session.publish(u'com.myapp.universal_feedback', {"id": context, "value": value})

@inlineCallbacks
def hub(thing):
    """ Main script. """
    do_things
    yield publish('output', 'some data for you')
    do_more_things
    yield publish('status', 'a progress message')
    do_even_more_things
    yield publish('status', 'some more data')
    do_all_the_things
    yield publish('other', 'something else')

Twisted - how to make lots of Python code non-blocking

2 Answers2