Twisted - Deferred, if its not asynchronous, then whats the point?

Question

I've been tasked with learning Twisted. I am also somewhat new to Python in general, but have used other modern programming languages.

In reading over Twisted documentation, I keep running into examples that are

• Not complete executable examples
• Run in one thread

Coming from other languages, when I use some asynchronous mechanism, there is usually another thread of execution while I carry out some manner of work, then I am notified when that work is completed, and I react to its results.

I do see that it has some built in asynchronous mechanisms, but none of them provide the user with a means to create custom CPU bound asynchronous tasks akin to 'Tasks' in C# or 'work' with boost::asio in C++ that would run in parallel to the main thread.

I see that Twisted provides a means to asynchronously wait on IO and do things in on the same thread while waiting, if we are waiting on:

• network reads and writes
• keyboard input

It also shows me how to:

• Do some manner of integration with GUI tool kits to make use of their event loop, but doesn't go into detail.
• Schedule tasks using reactor on a timer, but doesn't do that task in parallel to anything else

It talks about async/await, but that is for python 3 only, and I am using python 2.7

I figured the some manner of thread pooling must be built into the reactor, but then when I read about the reactor, it says that everything runs on the main thread in reactor.run().

So, I am left confused.

• What is the point of deferreds, creating a callback chain and reacting to the results, if we aren't running anything in parallel?
• If we are running asynchronous code, how are we making our own custom asynchronous functions? (see keyword async in C#)

In other languages, I might create an async task to count from 1 to 10, while on the main thread, I might count from 'a' to 'z' at the same time. When the the task is complete I would get notified via a callback on a thread from a threadpool. I'd have the option to sync up, if I wanted to, by calling some 'wait' method. While the definition of "asynchronous" only involves the posting of the task, the getting of the result, and the callback when its done....I've never seen it used without doing things in parallel.

Answer 1

I'll address your questions (and statements that seem confusing) one-by-one:

"Examples that are not complete"

Restating what I posted in the comments: see my two previous answers for complete examples ( https://stackoverflow.com/a/30399317/3334178 & https://stackoverflow.com/a/23274411/3334178 ) and go through Krondo's Twisted Introduction

You said you are discounting these because "The examples are the network code in twisted, which has the asynchronisity built in and hidden.". I disagree with that assertion and will explain this in the next section.

"Examples are not asynchronous"

When your talking about "asynchronous programming" in the vain of pythons twisted/tornado/asyncio (or Node.JS or C select/poll/kpoll) your talking about model/pattern of programming that allows the programmer shape their code so that parts of it can run while other parts are blocked (in almost all cases the blocking is caused by a part of the program having to wait for IO).

These libraries/languages will certainly have ways they can do threading and/or multiprocessing, but those are layers grafted on top of the async design - and if that's genuinely what you need (I.E. you have an exclusively CPU bound need) the async systems are going to be a bad choice.

Let's use your "hidden away" comment to get into this a bit more

"Network examples are asych, but the asynchronousity is built in and hidden away"

The fundamental element of the async design is that you write your code so it should never block for IO - You've been calling out network but really we are talking about network/disk/keyboard/mouse/sound/serial - anything that (for whatever reason) can run slower than the CPU (and that the OS has a file-descriptor for).

Also, there isn't anything really "hidden away" about how it functions - async programming always uses non-blocking (status checking / call-back) calls for any IO channel it can operate on. If you dig enough in the twisted codebase all the async logic is in plain sight (Krondo's tutorial is really good for giving examples of this)

Let me use the keyboard as an example.

In sync code, you would use an input or a read - and the program would pause waiting for that line (or key) to be typed.

In async code (at least in featureful implementations like twisted) you will fetch the file-descriptor for "input" and register it with call-back function, to be called when the file-descriptor changes, to the OS-level async engine (select, poll, kpoll, etc...)

The act of doing that registration - which takes almost no time LETS YOU run other logic while the keyboard logic waits for the keyboard event to happen (see the stdio.StandardIO(keyboardobj,sys.stdin.fileno()) line from near the end of my example code in https://stackoverflow.com/a/30399317/3334178).

"[This] leads me to believe there is some other means to use deferreds with asynchronous"

deferreds aren't magic. They are just clever lists of function callback. There are numerous clever ways they can be chained together, but in the end, they are just a tool to help you take advantage of the logic above

"It also talks about async/await, that is for python 3 only, and I am using python 2.7"

async and await are just the python 3 way of doing what was done in python2 with @defer.inlineCallbacks and yield. These systems are shortcuts that rewire code so that to the reader the code looks and acts like sync code, but when its run the code is morphed into a "register a callback and move-on" flow

"when I read about the reactor, it says that everything runs on the main thread in reactor.run()"

Yes, because (as above) async is about not-waiting-for-IO - its not about threading or multi-processing

Your last few questions "point of deferreds" and "how do you make asynchronous" feel like I answered them above - but if not, let me know in the comments, and I'll spell them out.

Also your comment requesting "an example where we count from 1 to 10 in some deferred function while we count from a to z in the main thread?" doesn't make sense when talking about async (both because you talk about a "thread" - which is a different construct, and because those are both (likely) CPU tasks), but I will give you a different example that counts while watching for keyboard input (which is something that definitely DOES make sense when talking about async:

#!/usr/bin/env python
#
# Frankenstein-esk amalgam of example code
#   Key of which comes from the Twisted "Chat" example
#   (such as: http://twistedmatrix.com/documents/12.0.0/core/examples/chatserver.py)

import sys # so I can get at stdin
import os # for isatty
import termios, tty # access to posix IO settings
from twisted.internet import reactor
from twisted.internet import stdio # the stdio equiv of listenXXX
from twisted.protocols import basic # for lineReceiver for keyboard
from twisted.internet import task


class counter(object):
    runs = 0

def runEverySecond():
    counter.runs += 1
    print "async counting demo: " + str(counter.runs)

# to set keyboard into cbreak mode - so keys don't require a CR before causing an event
class Cbreaktty(object):
    org_termio = None
    my_termio = None

    def __init__(self, ttyfd):
        if(os.isatty(ttyfd)):
            self.org_termio = (ttyfd, termios.tcgetattr(ttyfd))
            tty.setcbreak(ttyfd)
            print '  Set cbreak mode'
            self.my_termio = (ttyfd, termios.tcgetattr(ttyfd))
        else:
          raise IOError #Not something I can set cbreak on!

    def retToOrgState(self):
        (tty, org) = self.org_termio
        print '  Restoring terminal settings'
        termios.tcsetattr(tty, termios.TCSANOW, org)


class KeyEater(basic.LineReceiver):

    def __init__(self):
        self.setRawMode() # Switch from line mode to "however much I got" mode

    def rawDataReceived(self, data):
        key = str(data).lower()[0]
        if key == 'q':
            reactor.stop()
        else:
            print "--------------"
            print "Press:"
            print "      q  - to cleanly shutdown"
            print "---------------"

# Custom tailored example for SO:56013998
# 
# This code is a mishmash of styles and techniques. Both to provide different examples of how
# something can be done and because I'm lazy.  Its been built and tested on OSX and linux,
# it should be portable (other then perhaps termal cbreak mode).  If you want to ask
# questions about this code contact me directly via mail to mike at partialmesh.com
#
#
# Once running press any key in the window where the script was run and it will give
# instructions.  


def main():

    try:
      termstate = Cbreaktty(sys.stdin.fileno())
    except IOError:
      sys.stderr.write("Error: " + sys.argv[0] + " only for use on interactive ttys\n")
      sys.exit(1)

    keyboardobj = KeyEater()

    l = task.LoopingCall(runEverySecond)
    l.start(1.0) # call every second

    stdio.StandardIO(keyboardobj,sys.stdin.fileno())
    reactor.run()
    termstate.retToOrgState()
if __name__ == '__main__':
  main()

(I know technically I didn't use a deferred - but I ran out of time - and this case is a bit too simple to really need it (I don't have a chain of callback anywhere, which is what deferreds are for))