Hypothesis, django, and generating external IDs

Question

This is the first time I have tried to use Hypothesis with Django.

I apologize in advance for only being able to share code snippets. This is a very small piece of an extensive code base.

I have a Model with (among lots of other fields) a 1:1 external reference:

class Widget(models.Model):
   external_id = models.IntegerField(null=False, unique=True, db_index=True)

I'm trying to put together a Strategy to generate these Widgets. Almost everthing I've tried fails with "hypothesis.errors.Flaky: Inconsistent data generation! Data generation behaved differently between different runs. Is your data generation depending on external state?"

This is part of a fairly complex test that builds a fairly complex tree of inter-related Models. Narrowing the failure down to this particular field in this particular Model took quite a bit of time. The actual exception happens when a hypothesis.internal.conjecture.ConjectureData instance calls self.observer.conclude_test() from inside its .freeze() method. In some variations, the call stack includes a tuple that matches my @given arguments, but the cutover from there to the specific model (much less the field) being generated doesn't have any state that seems useful for diagnostics.

The basic idea I'm trying is:

import hypothesis.extra.django as hy_dj
import hypothesis.strategies as st

@st.composite
def Strategy(draw):
   return draw(hy_dj.from_model(Widget),

Removing the field's unique=True constraint allows the test to pass, but it destroys one of our bedrock assumptions.

Specifying external_id=st.just(1) fails to produce valid examples.

Everything else I've tried fails with that consistency error.

st.just(1)
st.just(random.randint(MAX_INT32))
st.sampled_from(range(1,1000))

Questions:

What is a good way to define a strategy for this field?
For future reference, is there a good way, short of trial and error, to tell which field from which property failed?

Thank you.

score 1 · Answer 1 · answered Nov 13 '19 at 02:11

My best guess, without being able to see any of your code, is that you're not resetting the DB between Hypothesis examples - so the test is flaky from Hypothesis' perspective, because subsequent runs are against a dirty DB.

Plus:

@st.composite
def Strategy(draw):
   return draw(hy_dj.from_model(Widget))

# is exactly equivalent to
def Strategy():
    return hy_dj.from_model(Widget)

Hypothesis, django, and generating external IDs

1 Answers1