Doctest failed with zero exit code

Question

In my test code, my doctest fails but the script exits with a zero return value, which causes the CI run to pass, which is not intended.

Is this the correct behavior of doctest module?

My script ends with:

if __name__ == '__main__':
    import doctest
    doctest.testmod()

The output is like:

**********************************************************************
File "test/test.py", line 7, in __main__
Failed example:
    f(1,0)
Expected:
    -----
    type: <type 'exceptions.ZeroDivisionError'>
    value: integer division or modulo by zero
    x
    -----
Got:
    -----
    type: <type 'exceptions.ZeroDivisionError'>
    value: integer division or modulo by zero
    -----
**********************************************************************
1 items had failures:
   1 of   1 in __main__
***Test Failed*** 1 failures.
tux@iPad:~/programming/exception-notifier(fix-travis)(0)$ echo $?
0

Can you please post the full stacktrace ? – Paco Sep 13 '13 at 09:32 — Paco, Sep 13 '13 at 09:32
dropping the X from your docstring ? – Sep 13 '13 at 09:53 — , Sep 13 '13 at 09:53
That x is deliberately added to make the test fail. – Fish Monitor Sep 13 '13 at 10:02 — Fish Monitor, Sep 13 '13 at 10:02

score 7 · Answer 1 · answered Sep 05 '14 at 18:37

@fossilet's answer works for properly breaking builds that fail tests, but it raises the exception before doctest is able to write anything to the console. This makes your logs much less useful for identifying the problem.

An alternative is to call

sys.exit(doctest.testmod()[0])

This makes the process exit code equal to the number of tests that failed. Your CI tool should interpret nonzero exit codes as failing builds. But the doctest output will still make it to the console.

Fish Monitor · Accepted Answer · 2013-09-16T03:25:58.303

5

I find using doctest.testmod(raise_on_error=True) will cause an exception to be raised when a test fails, which causes the script exits with a non-zero code.

Python doc here:

Optional argument raise_on_error defaults to false. If true, an exception is raised upon the first failure or unexpected exception in an example. This allows failures to be post-mortem debugged. Default behavior is to continue running examples.

edited Sep 16 '13 at 03:25

answered Sep 13 '13 at 10:01

Fish Monitor

4,155
3
32
53

Unfortunately using `raise_on_error` leads to a missing test output, so I switched to `sys.exit` based solution. – geekQ Jun 19 '19 at 15:53
1

@geekQ I'm using `verbose` flag in combination with `raise_on_error` to produce output and this allows to figure out which test has failed: `doctest.testmod(verbose=True, raise_on_error=True)` – juggernaut Jun 20 '19 at 10:16

Doctest failed with zero exit code

2 Answers2