How do I make a long task non-blocking with urwid and asyncio?

Question

I am writing a Python curses application that controls an external (Linux, if that helps) process by sending and receiving strings through the process' stdin and stdout, respectively. The interface uses urwid. I have written a class to control the external process and a couple of others for a few urwid components.

I also have a button that is supposed to send a command to the external process. However the process will not respond immediately and its task usually takes up to a few seconds, during which I'd like the interface not to freeze.

Here's how I run the child process:

def run(self, args):
    import io, fcntl, os
    from subprocess import Popen, PIPE

    # Run wpa_cli with arguments, use a thread to feed the process with an input queue
    self._pss = Popen(["wpa_cli"] + args, stdout=PIPE, stdin=PIPE)
    self.stdout = io.TextIOWrapper(self._pss.stdout, encoding="utf-8")
    self.stdin = io.TextIOWrapper(self._pss.stdin, encoding="utf-8", line_buffering=True)

    # Make the process' stdout a non-blocking file
    fd = self.stdout.fileno()
    fl = fcntl.fcntl(fd, fcntl.F_GETFL)
    fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
    ...

I've had to make the process' output stream non blocking to be able to parse its output. I don't know if that's important for my question.

Here's are the methods I use to control the child process input and output streams:

def read(self, parser=None, transform=None, sentinel='>'):
    """ Reads from the controlled process' standard output until a sentinel
    is found. Optionally execute a callable object with every line. Parsed
    lines are placed in a list, which the function returns upon exiting. """

    if not transform:
        transform = lambda str: str

    def readline():
        return transform(self.stdout.readline().strip())

    # Keep a list of (non-empty) parsed lines
    items = []
    for line in iter(readline, sentinel):
        if callable(parser):
            item = parser(line)
            if item is not None:
                items.append(item)
    return items

def send(self, command, echo=True):
    """ Sends a command to the controlled process. Action commands are
    echoed to the standard output. Argument echo controls whether or not
    they're removed by the reader function before parsing. """

    print(command, file=self.stdin)

    # Should we remove the echoed command?
    if not echo:
        self.read(sentinel=command)

The button I talked about just has its callback set from the main script entry function. That callback is supposed to send a command to the child process and loop through the resulting output lines until a given text is found, in which case the callback function exits. Until then the process outputs some interesting info that I need to catch and display in the user interface.

For instance:

def button_callback():
    # This is just an illustration

    filter = re.compile('(event1|event2|...)')

    def formatter(text):
        try:
            return re.search(filter, text).group(1)
        except AttributeError:
            return text

    def parser(text):
        if text == 'event1':
            # Update the info Text accordingly
        if text == 'event2':
            # Update the info Text accordingly

    controller.send('command')
    controller.read(sentinel='beacon', parser=parser, transform=formatter)

What's to notice is that:

• the read() function spins (I couldn't find another way) even while the process output stream is silent until the sentinel value is read from the (optionally) parsed lines,
• the urwid interface won't refresh until the button callback function exits, which prevents urwid's main loop from refreshing the screen.

I could use a thread but from what I've read urwid supports asyncio and that's what I'd like to implement. You can call me dumb for I just can't clearly figure out how even after browsing urwid asyncio examples and reading Python asyncio documentation.

Given that there's room to alter any of those methods I still would like to keep the process controlling class — i.e. the one that contains read() and send() — as generic as possible.

So far nothing I have tried resulted in the interface being updated while the process was busy. The component that receives the process' "notifications" is a plain urwid.Text() widget.

Answer 1

Two things first:

• you don't necessarily need asyncio to do async stuff with urwid, because it already has a simple event loop which is often good enough, it has primitives for handling many IO scenarios.

• when writing async code, you need to beware of writing sync code like those functions looping until it finds a sentinel value, because it will block any other code (including the event loop itself) from executing: this means that the UI will freeze until that function returns

For your case, you can probably use the default simple event loop and make usage of the MainLoop.watch_pipe method, which creates a pipe ready to use in a subprocess (already putting it in async/non-blocking mode, btw :)) and takes a callback to call when there is new data written to the pipe.

Here is a simple example of using it, showing the output of a shell command while keeping the UI non-blocked (watch out, using some global variables because laziness):

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from __future__ import print_function, absolute_import, division
import subprocess
import urwid


def show_or_exit(key):
    if key in ('q', 'Q', 'esc'):
        raise urwid.ExitMainLoop()


def update_text(read_data):
    text.set_text(text.text + read_data)


def enter_idle():
    loop.remove_watch_file(pipe.stdout)


if __name__ == '__main__':
    widget = urwid.Pile([
        urwid.Button('Here is a button'),
        urwid.Button('And here another button'),
        urwid.Button('One more, just to be sure'),
        urwid.Button("Heck, let's add yet another one!"),
    ])
    text = urwid.Text('PROCESS OUTPUT:\n')
    widget = urwid.Columns([widget, text])
    widget = urwid.Filler(widget, 'top')

    loop = urwid.MainLoop(widget, unhandled_input=show_or_exit)
    stdout = loop.watch_pipe(update_text)
    stderr = loop.watch_pipe(update_text)
    pipe = subprocess.Popen('for i in $(seq 50); do echo -n "$i "; sleep 0.5; done',
                            shell=True, stdout=stdout, stderr=stderr)
    loop.run()

Note how the code in the callback update_text doesn't have reasons to block: it gets the data that was read, updates the component and that's it. No while looping waiting for something else to happen.

In your case, you will probably need to adapt your functions that parse the output of wpa_cli so that they also don't have reasons to block. For example, instead of waiting on a loop until finding the value, they could set some variable or otherwise signal, when they've found or not the interesting sentinel value.

I hope this make sense, let me know if you need clarification on something! :)