How can a python 2 doctest fail and yet have no difference in the values in the failure message?

Question

I'm using Python 2.7.9 in Windows.

I have a UTF-8-encoded python script file with the following contents:

# coding=utf-8

def test_func():
    u"""
    >>> test_func()
    u'☃'
    """
    return u'☃'

I get a curious failure when I run the doctest:

Failed example:
    test_func()
Expected:
    u'\u2603'
Got:
    u'\u2603'

I see this same failure output whether I launch the doctests through the IDE I usually use (IDEA IntelliJ), or from the command line:

> x:\my_virtualenv\Scripts\python.exe -m doctest -v hello.py

I copied the lines under Expected and Got into WinMerge to rule out some subtle difference in the characters I couldn't spot; it told me they were identical.

However, if I redo the command line run, but redirect the output to a text file, like so:

> x:\my_virtualenv\Scripts\python.exe -m doctest -v hello.py > out.txt

the test still fails, but the resulting failure output is a bit different:

Failed example:
    test_func()
Expected:
    u'☃'
Got:
    u'\u2603'

If I put the escaped unicode literal in my doctest:

# coding=utf-8

def test_func():
    u"""
    >>> test_func()
    u'☃'
    """
    return u'\\u2603'

the test passes. But as far as I can tell, u'\u2603' and u'☃' should evaluate to the same thing.

Really I have two questions about the failing case:

Is one of the representations that the doctester is giving (under Expected or Got) incorrect for the value that the doctester has for that case? (i.e. x != eval(repr(x)))
If not, why does the test fail?

To sum up the issue: the doctester compares the results in the representation domain. The issue is actually more like `x != repr(eval(x))` (which happens, as there is more than one way to represent the same string in Python); the doctester takes the representation of the actual function output, which has `\u` escape sequences, and compares it with the expected representation I've given it which has the literal unicode character. When that fails, it prints the representation using a format operator that also converts the literal unicode characters in the expected representation to escapes. — rakslice, Jun 03 '15 at 23:29
Having a test that compares the representations of the values rather than comparing the actual values isn't ideal, but it's probably more practical to implement. — rakslice, Jan 27 '17 at 23:46

Zach Gates · Accepted Answer · 2017-09-20T02:23:30.030

7

The doctest module uses difflib to differentiate between the result and the expected result. Like the following:

>>> import difflib
>>> variation = difflib.unified_diff('x', 'x')
>>> list(variation)
[]
>>> variation = difflib.unified_diff('x', 'y')
>>> list(variation)
['--- \n', '+++ \n', '@@ -1 +1 @@\n', '-x', '+y']

Under the hood, the doctest module formats the result and expected result several times. Your problem seems to be an interpretation mistake caused by the string encodings. What gets printed to the console has been formatted (using %s), thus getting rid of any visible differences; making them look identical.

edited Sep 20 '17 at 02:23

answered May 16 '15 at 08:10

Zach Gates

4,045
1
27
51

Okay, I made the mistaken assumption that the doctester would be comparing the return value with value of the expression given as the expected output, since that's probably what you care about when you are testing a function. Instead they are comparing the representations, which may not be ideal as a test, but simplifies the implementation (it's just a text diff), and lets them implement the doctest ellipsis mode (`# doctest: +ELLIPSIS` with `...` wildcard) – rakslice Jun 03 '15 at 23:20

keepAlive · Answer 2 · 2017-08-18T13:32:53.857

Just for free and also because this possibility is not considered in the working discussion: I had a weakly similar problem. See

[...]
Expected:
    <xarray.DataArray ()>
    array(0.0)
    Coordinates:
        d1   |S3 'nat'
        d2   |S3 'dat'
        d3   |S3 'a'        
Got:
    <xarray.DataArray ()>
    array(0.0)
    Coordinates:
        d1   |S3 'nat'
        d2   |S3 'dat'
        d3   |S3 'a'

For sure, no human-visible difference. The solution in my trivial case was to ensure there were no whitespace !

How can a python 2 doctest fail and yet have no difference in the values in the failure message?

2 Answers2

Linked