2

I'm new to hypothesis and tried this simple code:

@settings(max_examples=5) 
@given(st.integers(), st.integers(), st.integers(), st.integers()) 
def test_foo(a,b,c,d): 
    print(a,b,c,d)

As you can see, test_foo receives 4 different integers. I run foo 5 times This is the print results:

0 0 0 0
0 0 0 0
-2070532028 -5212 -20927 14943
0 0 0 0
0 0 0 0

4 of the 5 times this test ran, it ran with 4 zeros. This isn't accidental, Every time I run test_foo I get the same results: 3-4 runs with 4 zeros and 1-2 runs with non zeros

How can I get more variant parameters?

Georgy
  • 12,464
  • 7
  • 65
  • 73
UdiM
  • 480
  • 3
  • 19
  • 1
    [This](https://stackoverflow.com/questions/50337248/generating-unique-ids-that-are-not-repeated-in-hypothesis) question might help you. – Thomas Lang Apr 02 '20 at 18:21

1 Answers1

3

The only solution to this issue is to run (many) more than five examples!

Hypothesis doesn't guarantee that it will only produce any particular input once, and in fact we have to produce some duplicates to check for flaky tests. We also prefer to explore a variety of simple examples early on, when it's most efficient. The implementation details of why you see this particular pattern are complicated, so I'll just say that if you run with the default 100 examples it won't be a problem.

Zac Hatfield-Dodds
  • 2,455
  • 6
  • 19
  • also from my experience it may be a great thing to test some crucial parts with 10k or more examples at least once after big refactoring, because you will never know what kind of examples may cause a bug – Azat Ibrakov Apr 06 '20 at 07:41
  • I have an expensive test involving Spark where running the test 100s or 1000s of times just isn't feasible. Is there any option to increase the "randomness" in such cases? In the best case, Hypothesis would check how many examples it should run (in my case e.g. only 50) and cover as much of the input space as possible in those runs. – Nightscape Feb 10 '22 at 00:23
  • Not really - but if the first 10 of 50 examples are simpler I think you're still OK. Personally I'd be comfortable turning it down to 25 for slow tests in unit-test workflows, and then running an overnight job occasionally with a few thousand examples or using https://hypofuzz.com/ – Zac Hatfield-Dodds Feb 11 '22 at 02:39