Tornado doesnt yield futures correctly

Question

I need to asynchronously make several http calls inside a tornado request handler.

trying to return futures is not documented well and pretty much impossible to do on a gathered asyncio.gather at the handler level of tornado.

I have tried aiohttp which works fine by itself however when putting it inside a tornado handler it throws loop is already in use. If you can show me how to inject into the the IOLoop some new futures to resolve that would be nice.

I have also tried using tornados AsyncHTTPClient which contrary to the documentation does not actually use yield but returns the response when you use await on it.

Is there any up to date documentation on this? ALL the example do not work for multiple async requests.

according to this documentation http://www.tornadoweb.org/en/stable/gen.html#module-tornado.gen

@gen.coroutine
def get(self):
    http_client = AsyncHTTPClient()
     response1, response2 = yield [http_client.fetch(url1),
                              http_client.fetch(url2)]
    response_dict = yield dict(response3=http_client.fetch(url3),
                           response4=http_client.fetch(url4))
    response3 = response_dict['response3']
    response4 = response_dict['response4']

however when trying to do this myself yield throws an error, substituting that with await gets a result. however you cannot await on a dict object like yield can. How can i get around this?

python 3.6.7 tornado 5.1.1 aiohttp 3.5.4

If you want to send multiple http requests asynchronously, use [`gen.WaitIterator`](http://www.tornadoweb.org/en/stable/gen.html#tornado.gen.WaitIterator). It allows you to yield futures from a list. I answered a similar question some time ago: https://stackoverflow.com/a/54125927/1925257 — xyres, Jan 31 '19 at 18:10
@xyres why is the documentation i linked for tornado wrong then? it clearly shows yielding multiple futures from a list. baffling. on top of that it is a bad mask for what I said above anyway. you can create your own list of futures and then `await` them instead of yielding or the overcomplicated WaitIterator, like the badly made tornado docs try to tell you. — Erik K, Feb 01 '19 at 09:16
@xyres to add to that `WaitIterator` still throws the error `"AssertionError: yield from wasn't used with future\n"` Theres something inherently broken in tornados interaction with the broken asyncio — Erik K, Feb 01 '19 at 10:29
@ErikK First of all, calm down. Second, `WaitIterator` lets you yield/await futures asynchronously, which means it gives you the result of the future as they get resolved. Whereas if you just yield/await a list of futures, you will not get the result as the futures resolve. You will only get the result when **all** the futures are resolved. — xyres, Feb 01 '19 at 11:18
@ErikK Also, please don't come in here all guns blazing, calling well maintained libraries "broken" because you're having a hard time learning to use them. Good luck finding help with this attitude. — xyres, Feb 01 '19 at 11:23
@xyres lets start with your similar answer. you show it working outside a tornado handler, using asyncio. tornado pretends to use asyncio under the hood now. if you try to yield while in the handler it break. if you try WaitIterator while in the handler it breaks. if you try to call the current event loop while in the handler, it breaks. Would love to see your example work in a handler. its a pretty simple time to use it in node. — Erik K, Feb 01 '19 at 11:32

score 1 · Accepted Answer · answered Feb 02 '19 at 15:50

Your comments use the word await, so it sounds like you're running in to a difference between native coroutines (defined with async def and await) and decorated coroutines (defined with @gen.coroutine and yield). You can only yield lists and dictionaries directly in decorated coroutines.

In a native coroutine, you must use tornado.gen.multi (or asyncio.gather):

async def get(self):
    http_client = AsyncHTTPClient()
    response1, response2 = await gen.multi([http_client.fetch(url1),
                                            http_client.fetch(url2)])
    response_dict = await gen.multi(dict(response3=http_client.fetch(url3),
                                         response4=http_client.fetch(url4)))
    response3 = response_dict['response3']
    response4 = response_dict['response4']

The differences between the two styles of coroutines are documented here

Thanks for that. I must have missed the tiny bit somewhere that makes the distinction between native async and tornados coroutine. I already have a `asyncio.gather` working outside tornado handlers for multi futures. However, when plugging into tornado i got into loop already running errors or yield errors. I think your response has the answers i need though thanks — Erik K, Feb 03 '19 at 12:40

Tornado doesnt yield futures correctly

1 Answers1