How can I read stderr with python if I'm NOT using subprocess?

Question

I am working with the cwiid library, which is a library written in C, but used in python. The library allows me to use a Wiimote to control some motors on a robot. The code is running as a daemon on an embedded device without a monitor, keyboard, or mouse.

When I try to initialize the object:

import cwiid

while True:
    try:
        wm = cwiid.Wiimote()
    except RuntimeError:
        # RuntimeError exception thrown if no Wiimote is trying to connect

        # Wait a second
        time.sleep(1)

        # Try again
        continue

99% of the time, everything works, but once in a while, the library gets into some sort of weird state where the call to cwiid.Wiimote() results in the library writing "Socket connect error (control channel)" to stderr, and python throwing an exception. When this happens, every subsequent call to cwiid.Wiimote() results in the same thing being written to stderr, and the same exception being thrown until I reboot the device.

What I want to do is detect this problem, and have python reboot the device automatically.

The type of exception the cwiid library throws if it's in a weird state is also RuntimeError, which is no different than a connection timeout exception (which is very common), so I can't seem to differentiate it that way. What I want to do is read stderr right after running cwiid.Wiimote() to see if the message "Socket connect error (control channel)" appears, and if so, reboot.

So far, I can redirect stderr to prevent the message from showing up by using some os.dup() and os.dup2() methods, but that doesn't appear to help me read stderr.

Most of the examples online deal with reading stderr if you're running something with subprocess, which doesn't apply in this case.

How could I go about reading stderr to detect the message being written to it?

I think what I'm looking for is something like:

while True:
    try:
        r, w = os.pipe()
        os.dup2(sys.stderr.fileno(), r)

        wm = cwiid.Wiimote()
    except RuntimeError:
        # RuntimeError exception thrown if no Wiimote is trying to connect

        if ('Socket connect error (control channel)' in os.read(r, 100)):
            # Reboot

        # Wait a second
        time.sleep(1)

        # Try again
        continue

This doesn't seem to work the way I think it should though.

Oftentimes there is an error message with an exception. You can do `except RuntimeError as e:` and then read the error message with `str(e)` and react differently depending on the message. — SethMMorton, Dec 01 '16 at 00:28
After glancing over the library code on github, you may be out of luck. As it is right now, specific connection errors like the one you report, are only printed to `stderr` and then lumped together in a single `RuntimeError` with the message string of `"Error opening wiimote connection"`. What SethMMorton says is valid, but you probably won't find the stderr string out there. At least not yet. As per the original question, intercepting the `stderr` of another process, this might be too much, but worth to check: http://github.com/jerome-pouiller/reredirect Good luck. — sal, Dec 01 '16 at 00:45
Okay, I can show you how to redirect stderr within-process the way you want, but can you clarify something in your loop example: what do you even do with `wm`? It appears that if the call to cwiid.Wiimote() succeeds, you just re-loop immediately -- there's no `break` from your `while True`. — K. A. Buhr, Dec 01 '16 at 17:43
The rest of my program is inside that while loop, further down. I could probably refactor it if I didn't want to indent the entire program, but it works for me. — John, Dec 01 '16 at 18:04
Your posted solution doesn't work because you're duping stderr to the wrong end of the pipe. Once you fix that, it won't work because your read will block if there's nothing output to stderr (e.g., a successful Wii call). Once you fix that, you'll run into issues where the program mysteriously exits without printing any error messages (because stderr is still redirected), etc., etc. If you already read my answer below, you are probably thinking "there must be a simpler way!" but this may be a case where there isn't. — K. A. Buhr, Dec 01 '16 at 19:36

score 1 · Answer 1 · answered Dec 01 '16 at 00:49

1

As an alternative to fighting with stderr, how about the following which retries several times in quick succession (which should handle connection errors) before giving up:

while True:
    for i in range(50):  # try 50 times
        try:
            wm = cwiid.Wiimote()
            break        # break out of "for" and re-loop in "while"
        except RuntimeError:
            time.sleep(1)
    else:
        raise RuntimeError("permanent Wiimote failure... reboot!")

answered Dec 01 '16 at 00:49

K. A. Buhr

45,621
3
45
71

cwiid.Wiimote() can fail thousands of times. It's job is to fail forever, until someone picks up a wiimote and hits the button in order to make it connect. This would basically make the robot constantly reboot every x minutes just in case it's locked up. Works in theory, but not exactly the best solution. – John Dec 01 '16 at 15:15

K. A. Buhr · Accepted Answer · 2016-12-01T20:07:12.097

Under the hood, subprocess uses anonymous pipes in addition to dups to redirect subprocess output. To get a process to read its own stderr, you need to do this manually. It involves getting an anonymous pipe, redirecting the standard error to the pipe's input, running the stderr-writing action in question, reading the output from the other end of the pipe, and cleaning everything back up. It's all pretty fiddly, but I think I got it right in the code below.

The following wrapper for your cwiid.Wiimote call will return a tuple consisting of the result returned by the function call (None in case of RuntimeError) and stderr output generated, if any. See the tests function for example of how it's supposed to work under various conditions. I took a stab at adapting your example loop but don't quite understand what's supposed to happen when the cwiid.Wiimote call succeeds. In your example code, you just immediately re-loop.

Edit: Oops! Fixed a bug in example_loop() where Wiimote was called instead of passed as an argument.

import time

import os
import fcntl

def capture_runtime_stderr(action):
    """Handle runtime errors and capture stderr"""
    (r,w) = os.pipe()
    fcntl.fcntl(w, fcntl.F_SETFL, os.O_NONBLOCK)
    saved_stderr = os.dup(2)
    os.dup2(w, 2)
    try:
        result = action()
    except RuntimeError:
        result = None
    finally:
        os.close(w)
        os.dup2(saved_stderr, 2)
        with os.fdopen(r) as o:
            output = o.read()
    return (result, output)

## some tests

def return_value():
    return 5

def return_value_with_stderr():
    os.system("echo >&2 some output")
    return 10

def runtime_error():
    os.system("echo >&2 runtime error occurred")
    raise RuntimeError()

def tests():
    print(capture_runtime_stderr(return_value))
    print(capture_runtime_stderr(return_value_with_stderr))
    print(capture_runtime_stderr(runtime_error))
    os.system("echo >&2 never fear, stderr is back to normal")

## possible code for your loop

def example_loop():
    while True:
        (wm, output) = capture_runtime_stderr(cmiid.Wiimote)
        if wm == None:
            if "Socket connect error" in output:
                raise RuntimeError("library borked, time to reboot")
            time.sleep(1)
            continue
        ## do something with wm??

How can I read stderr with python if I'm NOT using subprocess?

2 Answers2