pytest capsys: checking output AND getting it reported?

Question

Python 3.4.1, pytest 2.6.2.

When a test fails, pytest will routinely report what was printed to stdout by the test. For instance this code:

def method_under_test():
    print("Hallo, Welt!")
    return 41

def test_result_only():
    result = method_under_test()
    assert result == 42

when executed as python -m pytest myfile.py, will report this:

================================== FAILURES ===================================
______________________________ test_result_only _______________________________

    def test_result_only():
        result = method_under_test()
>       assert result == 42
E       assert 41 == 42

pytestest.py:9: AssertionError
---------------------------- Captured stdout call -----------------------------
Hallo, Welt!
========================== 1 failed in 0.03 seconds ===========================

This is a very nice feature. But when I use pytest's built-in capsys fixture, like this:

def test_result_and_stdout(capsys):
    result = method_under_test()
    out, err = capsys.readouterr()
    assert out.startswith("Hello")
    assert result == 42

the report no longer contains the actual output:

================================== FAILURES ===================================
___________________________ test_result_and_stdout ____________________________

capsys = <_pytest.capture.CaptureFixture object at 0x000000000331FB70>

    def test_result_and_stdout(capsys):
        result = method_under_test()
        out, err = capsys.readouterr()
>       assert out.startswith("Hello")
E       assert <built-in method startswith of str object at 0x000000000330C3B0>('Hello')
E        +  where <built-in method startswith of str object at 0x000000000330C3B0> = 'Hallo, Welt!\n'.startswith

pytestest.py:14: AssertionError
========================== 1 failed in 0.03 seconds ===========================

I am not sure whether this behavior is according to specification; the pytest documentation says about readouterr: "After the test function finishes the original streams will be restored."

I have tried assuming capsys is a context manager and have called capsys.__exit__() just before the asserts. This would be an ugly solution, but at least a solution if it restored the output before my assertion. However, this only produces

AttributeError: 'CaptureFixture' object has no attribute '__exit__'

Next I looked into the CaptureFixture class source code and found a promising-looking method close (which calls some pop_outerr_to_orig() method), but calling capsys.close() in my test did not help either, it had no obvious effect at all.

How can I get pytest to report my outputs upon failure in a test using capsys?

Answer 1

You're seeing the correct behaviour, when using capsys.readouterr() you're consuming the captured output. Hence any output to stdout and stderr will no longer show up in the test report. But any new output which you create after this and do not consume will still be reported, so you can get the full output back in the report by simply writing it to the output streams once more:

def test_result_and_stdout(capsys):
    result = method_under_test()
    out, err = capsys.readouterr()
    sys.stdout.write(out)
    sys.stderr.write(err)
    assert out.startswith("Hello")
    assert result == 42

Answer 2

From the documentation the behavior seems correct: only after the test function (test_result_and_stdout) finishes the output streams will be restored, not after each readouterr call. I don't think that currently capsys supports also redirecting to the original streams besides capturing them, which seems to be what you want.

I would suggest to create an issue in the official repository and see what people have to say.

Answer 3

As @flub describes, when using capsys, outputs to stdout and stderr are captured and consumed, which is the intended behaviour of capsys. You can write the messages back to stdout and stderr after capturing them if you want to see the messages in the test error capture.

However, I find that a common pattern is to run a couple of commands in series as part of the same test, typically because of state changes after one command affecting the behaviour of the next. When testing verbose outputs, you'll want to test the against the contents of the output of a specific command in the test, not the whole history of outputs which you've running during the test. In order to isolate the output from a single command, you need to capture stdout/stderr before calling the command as well as afterwards. That way you capture and discard the previous stdout contents before printing to stdout for an isolated command.

Unfortunately, this doesn't sit well with sending the messages back out to stdout and stderr after capturing them, since if you do that they will pollute the output from the next command within the test.

My solution is to make a helper function, recapsys, that plays the role of capturing and immediately regurgitating the contents of stdout/stderr (like how @flub describes). In addition, recapsys accepts arbitrarily many previous captures as arguments and will regurgitate these before the current capture. With this, you can capture previous stdout/stderr content before calling the command under test, and then print them out after running the command.

The definition for recapsys, and an example usage is given below. (For a class-based implementation of recapsys, check out the class methods in my python template repository.)

# Contents of test_fn.py

import sys


def recapsys(capsys, *captures):
    capture_now = capsys.readouterr()
    for capture in captures + (capture_now,):
        sys.stdout.write(capture.out)
        sys.stderr.write(capture.err)
    return capture_now


def test_stdout(capsys):
    capture_pre = capsys.readouterr()  # Clear stdout
    msg = "To be, or not to be, that is the question:"
    print(msg)  # Execute method (verbose)
    capture_post = recapsys(capsys, capture_pre)  # Capture and then re-output
    assert capture_post.out == msg + "\n"  # Compare output to target

    capture_pre = capsys.readouterr()  # Clear stdout
    msg = "Whether 'tis nobler in the mind to suffer"
    print(msg)  # Execute method (verbose)
    capture_post = recapsys(capsys, capture_pre)  # Capture and then re-output
    assert capture_post.out.lower().startswith("whether")  # Test

    capture_pre = capsys.readouterr()  # Clear stdout
    msg = "The slings and arrows of outrageous fortune,"
    print(msg)  # Execute method (verbose)
    capture_post = recapsys(capsys, capture_pre)  # Capture and then re-output
    assert capture_post.out == "We're no strangers to love\n"  # Test

Running pytest on test_fn.py produces the following output:

===================================== FAILURES ======================================
____________________________________ test_stdout ____________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x7f49015ce278>

    def test_stdout(capsys):
        capture_pre = capsys.readouterr()  # Clear stdout
        msg = "To be, or not to be, that is the question:"
        print(msg)  # Execute method (verbose)
        capture_post = recapsys(capsys, capture_pre)  # Capture and then re-output
        assert capture_post.out == msg + "\n"  # Compare output to target
    
        capture_pre = capsys.readouterr()  # Clear stdout
        msg = "Whether 'tis nobler in the mind to suffer"
        print(msg)  # Execute method (verbose)
        capture_post = recapsys(capsys, capture_pre)  # Capture and then re-output
        assert capture_post.out.lower().startswith("whether")  # Test
    
        capture_pre = capsys.readouterr()  # Clear stdout
        msg = "The slings and arrows of outrageous fortune,"
        print(msg)  # Execute method (verbose)
        capture_post = recapsys(capsys, capture_pre)  # Capture and then re-output
>       assert capture_post.out == "We're no strangers to love\n"  # Test
E       assert 'The slings a...us fortune,\n' == "We're no strangers to love\n"
E         - We're no strangers to love
E         + The slings and arrows of outrageous fortune,

test_fn.py:30: AssertionError
------------------------------- Captured stdout call --------------------------------
To be, or not to be, that is the question:
Whether 'tis nobler in the mind to suffer
The slings and arrows of outrageous fortune,
============================== short test summary info ==============================
FAILED test_fn.py::test_stdout - assert 'The slings a...us fortune,\n' == "We're n...
================================= 1 failed in 0.26s =================================

Answer 4

Asides using "startswith".You can also use the "in" keyword, for example:

    assert "Hello" in output

This is great if you have a huge amount of data being passed to stdout, you can use "in" to check for different lines in your stdout.

def test_result_and_stdout(capsys):
    result = method_under_test()
    out, err = capsys.readouterr()
    sys.stdout.write(out)
    sys.stderr.write(err)
    assert "Hello" in out 
    assert result == 42

You can also assert what's passed into stderr instead of stdout by using:

    assert "What you are expecting" in err

Also note that line:

    out, err = capsys.readouterr()

creates a snapshot of the output to both stdout and stderr so far so you can assert what you are expecting for that test.

Answer 5

In pytest >= 3, capsys and capsysfd both have disabled() methods that one can use to temporarily disable capturing, enabling one to output diagnostics. (As well as being cleaner, this is the only thing that worked for me.)