How to use Asynchronous Generators in Twisted?

Question

Twisted provides wrapping native coroutines(coroutines using async/await). Using ensureDeferred, we can wrap a coroutine and get its equivalent deferred. How to wrap an asynchronous generator (available in Python 3.6), so that we get a deferred ?

Trying to wrap an asynchronous object returns following error:

File "/home/yash/Scrapy/vnv/lib/python3.6/site-packages/Twisted-17.9.0-py3.6-linux-x86_64.egg/twisted/internet/defer.py", line 915, in ensureDeferred
raise ValueError("%r is not a coroutine or a Deferred" % (coro,))

ValueError: is not a coroutine or a Deferred

EDIT: I have provided a sample code where the logic required is demonstrated,and the corresponding error occuring:

from twisted.internet import reactor, defer
import asyncio


async def start_request():
   urls = ['https://example.com/page/1',
           'https://example.com/page/2',
           'https://example.com/page/3']

   async for url in urls:
       yield url
       await asyncio.sleep(2)


async def after_request():
   await asyncio.sleep(3)
   print("Completed the work")


deferred = ensureDeferred(start_request())
deferred.addCallback(after_request())
print("reactor has started")
reactor.run()

Error Traceback:

Traceback (most recent call last):
File "req2.py", line 20, in <module>
deferred = defer.ensureDeferred(start_request())
File "/home/yash/Scrapy/vnv/lib/python3.6/site-packages/Twisted-
17.9.0-py3.6-linux-x86_64.egg/twisted/internet/defer.py", line 915, in 
ensureDeferred
raise ValueError("%r is not a coroutine or a Deferred" % (coro,))
ValueError: <async_generator object start_request at 0x7febb282cf50> 
is not a coroutine or a Deferred

@Jean-Paul Calderone Thanks for the suggestions. I will add one. — hitman23, Mar 14 '18 at 02:47
You're getting this error because you're using yield inside a coroutine, which turn it into a generator function, not an actual coroutine. Also, urls is not an async iterator, so you shouldn't use async for with it. — Antonio Dourado, Mar 22 '18 at 18:06

score 0 · Answer 1 · answered Mar 22 '18 at 19:40

There's a couple of problems with the example you provided. The first one is causing the error you're seeing, and the other will cause problems later

The first one is that if you use yield inside a async function, it's not a coroutine anymore, it's a generator function. So, what you're passing to ensureDeferred is not a coroutine, which causes the error you're seeing. Instead of yield url, try to just print, or collect them in a list and return it later.

The second problem is that you can use async/await with twisted, but not asyncio, directly. Twisted code that wants to call asyncio functions must convert asyncio Futures to Deferreds first. Since asyncio.sleep gives you a coroutine, you need to convert that to a Future first, with asyncio.ensure_future, then convert the Future to a Deferred, with Deferred.fromFuture. Or you can just use twisted to simulate a sleep, like this:

def sleep(secs):
    d = Deferred()
    reactor.callLater(secs, d.callback, None)
    return d

The third problem is that you can't use async for with something that is not an async iterator, so the async for url in urls: should be just for url in urls:

P.S: Thanks to @runciter and @jfhbrook from the #twisted channel on Freenode for the help with this =)

How to use Asynchronous Generators in Twisted?

1 Answers1