3

I have seen several question on this subject, but no answer matching exactly what I want to do.

What I want to do with Pytest: don't report tests where an error occurred (any kind of error, ZeroDivision for example) as a failure, but as an error. The reason is simple: I want to know which tests highlight a possible bug and which tests didn't execute as expected for any other reason than a bug (bad configuration, network issue,...).

In this answer, test calling fixture is reported as an error, which is good. But I don't know which tests will cause an error, so how to do? If I call the fixture, the test will automatically be reported with an error.

I also tried to use a wrapper function as explained here but I can't make it work in Pytest.

To summarize, what I want to do is following:


class TestErrorVsFailure:
    def test_fail(self):
        print("Test expected to report a failure")
        assert 2 == 1

    def test_error(self):
        print("Test expected to report an error")
        a = 5/0

Result of Pytest execution: both tests are reported as failed

================================== FAILURES ===================================
________________________ TestErrorVsFailure.test_fail _________________________

self = <test_programs_services.TestErrorVsFailure object at 0x00000125F0633640>

    @pytest.mark.poc
    def test_fail(self):
        print("Test expected to report a failure")
>       assert 2 == 1
E       assert 2 == 1

tests\scenario\web\test_programs_services.py:43: AssertionError
________________________ TestErrorVsFailure.test_error ________________________

self = <test_programs_services.TestErrorVsFailure object at 0x00000125F066A490>

    @pytest.mark.poc
    def test_error(self):
        print("Test expected to report an error")
>       a = 5 / 0
E       ZeroDivisionError: division by zero

tests\scenario\web\test_programs_services.py:48: ZeroDivisionError

================= 2 failed in 1.48s =================

I can also try to catch as much as possible all exceptions, but raising a custom exception results as a failed test too so it seems not to be the right solution...

Thanks in advance if you have the solution!

UPDATE:

Pytest-html report shows 2 tests Failed too, as output of Pytest.

However, allure-pytest html report shows 1 test Failed and 1 test broken, which is what I want.

I am still interested in your feedbacks if you have any.

Francky
  • 31
  • 3
  • This is not the issue with pytest but with pytest-html. Save the result in junit-xml format and inspect it. It should mention broken there. – SilentGuy Oct 24 '20 at 00:23

0 Answers0