5

This question is different than Is there a way to use asyncio.Queue in multiple threads?

I have 2 asyncio event loops running in two different threads. Thread1 produces data to Thread2 through a asyncio.Queue().

One of the threads throws an exception: got Future <Future pending> attached to a different loop

Now this is true because I have a single queue that I use in different loops. How can I share a queue between two loops, in two different threads?

Example code:

q = asyncio.Queue()

async def producer(q):
    await asyncio.sleep(3)
    q.put(1)

def prod_work(q):
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    loop.run_until_complete(producer(q))

async def consumer(q):
    await asyncio.sleep(3)
    res = await q.get()

def cons_work(q):
    loop2 = asyncio.new_event_loop()
    asyncio.set_event_loop(loop2)
    loop2.run_until_complete(consumer(q))

def worker(q):
    # main thread - uses this threads loop  
    prod = threading.Thread(target=prod_work, args=(q,))

    # separate thread - uses NEW loop
    cons = threading.Thread(target=cons_work, args=(q,))

    prod.start()
    cons.start()

worker(q)

full stack trace:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "asyncio_examples.py", line 24, in cons_work
    loop2.run_until_complete(consumer(q))
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 568, in run_until_complete
    return future.result()
  File "asyncio_examples.py", line 18, in consumer
    res = await q.get()
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/queues.py", line 159, in get
    await getter
RuntimeError: Task <Task pending coro=<consumer() running at asyncio_examples.py:18> cb=[_run_until_complete_cb() at /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py:150]> got Future <Future pending> attached to a different loop
Chen A.
  • 10,140
  • 3
  • 42
  • 61
  • 1
    Why do you have two event loops running at the same time? Such setup can work, but asyncio was not designed to support it (and it's not needed), so you're rowing upstream to make it work. – user4815162342 Sep 24 '18 at 07:28
  • I have on thread that listens for new events; its part of the infra. My main thread gets messages from that one, and process them and dispatch some task back to the other thread. – Chen A. Sep 24 '18 at 07:37
  • The idiomatic approach is to run the event loop in a separate thread and use `run_coroutine_threadsafe` to submit coroutines to it. See e.g. [here](https://stackoverflow.com/a/50522178/1600898) for details. – user4815162342 Sep 24 '18 at 07:56

0 Answers0