I have seen several question on this subject, but no answer matching exactly what I want to do.
What I want to do with Pytest: don't report tests where an error occurred (any kind of error, ZeroDivision for example) as a failure, but as an error. The reason is simple: I want to know which tests highlight a possible bug and which tests didn't execute as expected for any other reason than a bug (bad configuration, network issue,...).
In this answer, test calling fixture is reported as an error, which is good. But I don't know which tests will cause an error, so how to do? If I call the fixture, the test will automatically be reported with an error.
I also tried to use a wrapper function as explained here but I can't make it work in Pytest.
To summarize, what I want to do is following:
class TestErrorVsFailure:
def test_fail(self):
print("Test expected to report a failure")
assert 2 == 1
def test_error(self):
print("Test expected to report an error")
a = 5/0
Result of Pytest execution: both tests are reported as failed
================================== FAILURES ===================================
________________________ TestErrorVsFailure.test_fail _________________________
self = <test_programs_services.TestErrorVsFailure object at 0x00000125F0633640>
@pytest.mark.poc
def test_fail(self):
print("Test expected to report a failure")
> assert 2 == 1
E assert 2 == 1
tests\scenario\web\test_programs_services.py:43: AssertionError
________________________ TestErrorVsFailure.test_error ________________________
self = <test_programs_services.TestErrorVsFailure object at 0x00000125F066A490>
@pytest.mark.poc
def test_error(self):
print("Test expected to report an error")
> a = 5 / 0
E ZeroDivisionError: division by zero
tests\scenario\web\test_programs_services.py:48: ZeroDivisionError
================= 2 failed in 1.48s =================
I can also try to catch as much as possible all exceptions, but raising a custom exception results as a failed test too so it seems not to be the right solution...
Thanks in advance if you have the solution!
UPDATE:
Pytest-html report shows 2 tests Failed too, as output of Pytest.
However, allure-pytest html report shows 1 test Failed and 1 test broken, which is what I want.
I am still interested in your feedbacks if you have any.