1

In spite of specifying maxfail=1, hypothesis seems to continue generating examples and running them and failing much later.

Is there a workaround?

Here is a small example:

from hypothesis.stateful import invariant, rule, RuleBasedStateMachine


class MaxFail(RuleBasedStateMachine):
    count = 0

    @rule()
    def process(self):
        self.count += 1

    @invariant()
    def all_done(self):
        print('-- in invariant %d' % self.count)
        if self.count > 1:
            assert False


MaxFailTest = MaxFail.TestCase
sureshvv
  • 4,234
  • 1
  • 26
  • 32
  • 2
    can you provide an example of test with expected and actual behavior? – Azat Ibrakov Apr 21 '20 at 15:34
  • I have added the example. When you run it, you will see invariant 2 printed several times. – sureshvv Apr 22 '20 at 07:47
  • 1
    That makes sense to me: any time you're running a MaxFailTest, hypothesis will run the rule, it wil process() then will check the invariant, and if the invariant fails then that instance is marked as a failure. Furthermore it's not clear that the invariant *stops* the generation process, I'd expect they don't and just mark this specific state is considered invalid. `@precondition` seems like it'd be more suitable at least for this trivial case. – Masklinn Apr 22 '20 at 07:57
  • AFAIK after failing example is found Hypothesis tries to simplify it, so you can have some pretty-looking and simple object in the end of this process – Azat Ibrakov Apr 22 '20 at 08:18
  • An assertion failure is a test failure. And when --maxfail=1 the run should stop after the 1st such failure. No? – sureshvv Apr 22 '20 at 11:31
  • I doubt that a `precondition` will help. Because it continues to run new examples. But will check and report. – sureshvv Apr 22 '20 at 11:32
  • Just as I suspected, adding a `precondition` of `self.count < 2` did not help. – sureshvv Apr 22 '20 at 11:35

1 Answers1

2

This is because from Pytest's point of view the whole stateful test is only one test - it invokes MaxFailTest.runTest(), and if that fails it will not run any other test functions.

On the other hand, Hypothesis doesn't know about any Pytest arguments or settings beyond those added in its plugin. It can be used equally well with pytest, or unittest, or any other test runner because it simply wraps the inner test function you wrote.

In short: Hypothesis doesn't know about the --maxfail argument, and Pytest doesn't know the test will fail until Hypothesis raises an error with the minimal example(s) it found.

Zac Hatfield-Dodds
  • 2,455
  • 6
  • 19
  • one test or more should not matter. maxfail means quit after so many failures. Can happen within same test. Would be good if Hypothesis honored this. Especially since they have gone to great lengths to integrate with pytest off the box. – sureshvv Apr 24 '20 at 07:13
  • The current behaviour is actually the best we can do, because the respective settings are for *test cases* (or *examples*) on the Hypothesis side, and *tests* on the Pytest side. Parametrize is a concise way to write many tests, while Hypothesis creates single tests which execute many examples. Source: I'm a pytest maintainer and Hypothesis core developer :-) – Zac Hatfield-Dodds Apr 24 '20 at 16:32