Getting .Flaky: Inconsistent data generation! in Python hypothesis

Question

My test is to create a policy using (API), but API does not allow duplicate rule name. so hypothesis when retries API is bound to give Duplicate name not allowed.

If add logic in my code to delete the policy first if same rule name is generated then I hit

hypothesis.errors.Flaky: Inconsistent data generation! Data generation behaved differently between different runs. Is your data generation depending on external state?

import string
from hypothesis import assume
import pytest
import unittest
from hypothesis import strategies as st, reject
from hypothesis.stateful import RuleBasedStateMachine, precondition, rule


used_names = set()


@st.composite
def unique_name_strategy(draw):
    name_strategy = st.text(min_size=5, max_size=31, alphabet=string.ascii_letters + string.digits)
    while True:
        name = draw(name_strategy)
        if name not in used_names:
            used_names.add(name)
            return name


class TestsEdlp(RuleBasedStateMachine):
    def __init__(self) -> None:
        super().__init__()

    rule_names = []
    tries = 0


    def rule_creation(self, rule_name):
        print("in rule creation")

        if rule_name in self.rule_names:
            self.rule_names.remove(rule_name)

        if rule_name:
            self.rule_names.append(rule_name)
            self.tries += 1
            return True
        else:
            return False


    @precondition(lambda self: self.tries < 10)
    @rule(rule_name=unique_name_strategy())
    def create_dlp_rule(self, rule_name):

        print(f"in function create rule {rule_name} tries: {self.tries}")
        is_success = self.rule_creation(rule_name=rule_name)
        assert is_success

TestsEdlp: unittest.TestCase = TestsEdlp.TestCase

score 0 · Answer 1 · answered Jul 04 '23 at 00:51

The problem is that your strategy for unique names doesn't allow Hypothesis to generate the same name in two different runs of your test function, which must be allowed - otherwise, we can't try variations or discover a minimal failing example (among other problems).

This may also require some work to reset the state of your API for each input Hypothesis tries. (as your later comments on this issue suggest)

A better unique-names strategy would be:

@st.composite
def unique_name_strategy(draw, name_strategy=st.text(string.ascii_letters + string.digits, min_size=5, max_size=31)):
    used_names = draw(st.shared(st.builds(set), key="used names"))
    name = draw(name_strategy.filter(lambda x: x not in used_names)
    used_names.add(name)
    return name

The st.shared() strategy will only generate one set for each input, so we get just the right amount of persistence.

Getting .Flaky: Inconsistent data generation! in Python hypothesis

1 Answers1