1

Using pytest, I set specific parameters to fail as expected.

My testing suite uses hypothesis though. These is a bug that we know about that we don't intend to fix in this version of the API. It's already documented and the time it will takes to fix it is not worth the effort required at this time.

Helpfully, hypothesis points out this failure everytime the test suite runs and we've learned to ignore it. Obviously this is bad (especially because it's leading us to ignore other things that we shouldn't).

Is there a way to tell pytest to expect a hypothesis failure for a single parameter value? I don't want to remove it completely because there is a larger project on the horizon that should fix it. When that occurs, I want hypothesis to check this condition again.

A test case I want to apply this do is defined like so:

@given(inp=st.text(min_size=1))
def test_input(inp):
   ...

My goal is to be able to say that when inp equals a specific value, expect a failure. Otherwise, this test should pass.

NewGuy
  • 3,273
  • 7
  • 43
  • 61
  • 1
    Why not just mark the test with `mark.xfail` (with an informative `reason`)? If `hypothesis` selects the failing example, the test will `xfail` as a single entity (there are no multiple tests per example anyway), `xpass`ing if the example was sorted out. – hoefling Jul 09 '19 at 08:21

1 Answers1

0

You can conditionally xfail (or similarly skip) the iteration

@given(inp=st.text(min_size=1))
def test_input(inp):
    if inp == some_value:
        pytest.xfail("Marking xfail as inp equals some_value")
Amit
  • 19,780
  • 6
  • 46
  • 54
  • 2
    As a developer of both Hypothesis and pytest, unfortunately that's not how the interaction works - you can check by running it with the arguments `-s --hypothesis-verbosity=verbose`; instead of skipping or xfailing that it raises a special exception and Hypothesis shrinks it as usual. Much more efficient to just use `@pytest.mark.xfail` as a decorator. – Zac Hatfield-Dodds Jul 12 '19 at 05:24
  • @ZacHatfield-Dodds - Thank you for this information. – Amit Jul 12 '19 at 11:45