12

I'm designing an application in Python which should access a machine to perform some (lengthy) tasks. The asyncio module seems to be a good choice for everything that is network-related, but now I need to access the serial port for one specific component. I've implemented kind of an abstraction layer for the actual serial port stuff, but can't figure out how to sensibly integrate this with asyncio.

Following setup: I have a thread running a loop, which regularly talks to the machine and decodes the responses. Using a method enqueue_query(), I can put a query string into a queue, which will then be sent to the machine by the other thread and cause a response. By passing in a threading.Event (or anything with a set() method), the caller can perform a blocking wait for the response. This can then look something like this:

f = threading.Event()
ch.enqueue_query('2 getnlimit', f)
f.wait()
print(ch.get_query_responses())

My goal is now to put those lines into a coroutine and have asyncio handle this waiting, so that the application can do something else in the meantime. How could I do this? It would probably work by wrapping the f.wait() into an Executor, but this seems to be a bit stupid, as this would create a new thread only to wait for another thread to do something.

Neuron
  • 5,141
  • 5
  • 38
  • 59
Philipp Burch
  • 869
  • 1
  • 7
  • 16

3 Answers3

21

By passing in a threading.Event (or anything with a set() method), the caller can perform a blocking wait for the response.

Given the above behavior of your query function, all you need is a thread-safe version of asyncio.Event. It's just 3 lines of code:

import asyncio
class Event_ts(asyncio.Event):
    #TODO: clear() method
    def set(self):
        #FIXME: The _loop attribute is not documented as public api!
        self._loop.call_soon_threadsafe(super().set)

A test for functionality:

def threaded(event):
    import time
    while True:
        event.set()
        time.sleep(1)


async def main():
    import threading
    e = Event_ts()
    threading.Thread(target=threaded, args=(e,)).start()
    while True:
        await e.wait()
        e.clear()
        print('whatever')


asyncio.ensure_future(main())
asyncio.get_event_loop().run_forever()
Huazuo Gao
  • 1,603
  • 14
  • 20
  • 4
    This should work fine in most cases, but it's worth noting that calling `e.set()` from the event loop thread will actually delay the execution of the `set()` call until the calling method either yields/awaits or exits altogether. Usually that's fine, but it might occasionally cause unexpected behavior. – dano Oct 08 '15 at 19:21
  • Solved my issue. Before I was trying to set an asyncio event in a thread that captures the keyboard input and raises an event every time the user enters something, but the event.wait() did not get it and was waiting forever. By replacing it with your Event_ts it's working fine , that is continue to execute once the event is set by the other thread. – Marco Jun 10 '20 at 08:08
  • THAT Event_ts IS SUPERB! – Dee Feb 25 '21 at 08:18
7

The simplest way is to do exactly what you suggested - wrap the call to f.wait() in an executor:

@asyncio.coroutine
def do_enqueue():
    f = threading.Event()
    ch.enqueue_query('2 getnlimit', f)
    yield from loop.run_in_executor(None, f.wait)
    print(ch.get_query_responses())

You do incur the overhead of starting up a thread pool (at least for the first call, the pool will stay in memory from that point forward), but any solution that provides an implementation of something like threading.Event() with thread-safe blocking and non-blocking APIs, without relying on any background threads internally, would be quite a bit more work.

dano
  • 91,354
  • 19
  • 222
  • 219
  • Thanks for your answer! Sounds reasonable to do it this way, I think I will use this one as well as the way suggested by Huazuo Gao, depending on the use-case. – Philipp Burch Oct 08 '15 at 17:53
  • 2
    Be aware though, that the default executor comes with `max_workers` (i.e. max coroutines waiting for an event) set to **40**. This bit me when I used the presented solution to make a `threading.barrier` awaitable. – Tom Pohl Jul 20 '19 at 20:55
6

The class Event_ts in Huazuo Gao's answer works well in Python up to 3.9 but not in 3.10. It is because in Python 3.10 the private attribute _loop is initially None.

The following code works in Python 3.10 as well as 3.9 and below. (I added the clear() method as well.)

import asyncio
class Event_ts(asyncio.Event):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        if self._loop is None:
            self._loop = asyncio.get_event_loop()

    def set(self):
        self._loop.call_soon_threadsafe(super().set)

    def clear(self):
        self._loop.call_soon_threadsafe(super().clear)
Tai Sakuma
  • 91
  • 2
  • 3