24

In Tornado, we usually write the following code to call a function asynchronously:

class MainHandler(tornado.web.RequestHandler):

    @tornado.gen.coroutine
    def post(self):
        ...
        yield self.handleRequest(foo)
        ...

    @tornado.gen.coroutine
    def handleRequest(self, foo):
        ...

But in asyncio (will be shipped with Python 3.4, can be installed from pip for Python 3.3), we use yield from to achieve the same thing:

@asyncio.coroutine
def myPostHandler():
    ...
    yield from handleRequest(foo)
    ...


@asyncio.coroutine
def handleRequest(foo)
    ...

Seeing from the code, the difference is yield and yield from. However the former handleRequest(foo) returns a tornado.concurrent.Future object, the latter returns a generator object.

My question is, what is the difference between the two things in mechanism? How is the control flow? And who calls the actual handleRequest and retrieves its returning value?

Append: I have basic knowledge of Python generators and iterators. I wanted to understand what Tornado and asyncio achieved by using these, and what is the difference between those two mechanisms.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
Star Brilliant
  • 2,916
  • 24
  • 32

1 Answers1

22

There is a huge difference between the two. yield from takes another generator and continues yielding from that generator instead (delegating responsibility, as it were). yield just yields one value.

In other words, yield from, in the simplest case, could be replaced by:

for value in self.handleRequest(foo):
    yield value

If you replaced a yield from <expression> line with yield <expression> you'd return the whole generator to the caller, not the values that generator produces.

The yield from syntax was only introduced in Python 3.3, see PEP 380: Syntax for Delegating to a Subgenerator. Tornado supports Python versions 2.6, 2.7 and 3.2 in addition to Python 3.3, so it cannot rely on the yield from syntax being available. asyncio, on the other hand, being a core Python library added in 3.4, can fully rely on the yield from generator delegation syntax being available.

As a result, Tornado will have to post-process values yielded from a @tornado.gen.coroutine generator to detect that a tornado.concurrent.Future object was yielded; the @asyncio.coroutine code handling can be much simpler. And indeed the Tornado Runner.run() method does explicit type checks to handle delegated tasks.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • Thank you for your quick reply. But my question is not related to `yield` itself. I am concerned about what Tornado and asyncio achieved by using `yield` and `yield from`. – Star Brilliant Jan 09 '14 at 14:04
  • @StarBrilliant: They achieve deferring the task until later; generators can be paused so that the event loop can pass control to another generator. – Martijn Pieters Jan 09 '14 at 14:40
  • @StarBrilliant: The coroutine runner will then loop through the active coroutines and give each a chance to execute. A coroutine that needs to wait for network resources can then just immediately yield again to allow control to pass to other coroutines that perhaps do not have to wait. – Martijn Pieters Jan 09 '14 at 14:50
  • 2
    @StarBrilliant: But your question appeared to be about the syntax difference; `yield from` makes the runner code simpler; `tornado` has to use explicit wrappers to handle delegation, `asyncio` can rely on the syntax to take care of delegation instead. – Martijn Pieters Jan 09 '14 at 14:51
  • Sorry about my unclear language expression. I am very satisfied with your answer now. Thank you a lot. – Star Brilliant Jan 09 '14 at 15:12
  • 4
    In addition to making the runner simpler, `yield from` is also faster and tends to result in better stack traces when there is an error. On the other hand, Tornado's use of `yield` provides better interoperability between coroutines and non-coroutines (including both asynchronous logic with explicit callbacks and threaded executors). – Ben Darnell Jan 09 '14 at 18:02