Hypothesis library: strategy for the complement of some other strategy

Question

I'm using the Hypothesis library for unit testing. With this library you don't hand-pick inputs, but you define the complete set of inputs for which you want to test. Hypothesis will then sample from this set to look for inputs that break the function. This is also known as property-based testing. In Hypothesis, these sets are called strategies.

Now I would like to unit test a function that validates some input:

GRIDSIZE_FORMAT = "^[0-9]+x[0-9]+$"
CELL_FORMAT = "^[A-Z]+[0-9]+$"

def _validate(gridsize, walls, entrance):
    if not re.match(GRIDSIZE_FORMAT, gridsize):
        raise ValueError(f"grid size '{gridsize}' does not match format '{GRIDSIZE_FORMAT}'")
    for wall in walls:
        if not re.match(CELL_FORMAT, walls):
            raise ValueError(f"wall '{wall}' does not match format '{CELL_FORMAT}'")
    if not re.match(CELL_FORMAT, entrance):
        raise ValueError(f"entrance '{entrance}' does not match format '{CELL_FORMAT}'")

To properly test this function I want to generate examples of the form "anything but X", X being the proper input format for this function.

Is there a strategy in the Hypothesis library that can generate inputs like this?

Zac Hatfield-Dodds · Accepted Answer · 2022-04-25T01:20:21.170

Unfortunately Hypothesis cannot compute the complement of a strategy, because strategies can be composed with arbitrary user-supplied code (including side-effects!). For example, what should be the complement of hypothesis.extra.django.from_model(User), which inserts the generated instance into the database? (invalid, not in the database, ...)

In more specific cases, you can instead take the complement of your specification and then derive a strategy from that complement instead. Your regex trick is a good example - explicit set-complement tends to make this more efficient than the naive "generate something right-ish, and filter out valid instances" approach.

score 0 · Answer 2 · answered Apr 14 '22 at 11:27

I did find a way to do this with regexes:

from hypothesis import strategies as st

not_cellpattern = st.from_regex(f'(?!{CELL_FORMAT})')
@given(
    gridsize = st.from_regex(f'(?!{GRIDSIZE_FORMAT})'),
    walls = st.lists(not_cellpattern),
    entrance = not_cellpattern
)
def test_validate(gridsize, walls, entrance):
    try:
        _validate(gridsize, walls, entrance)
    except ValueError:
        pass
    else:
        raise Exception(f"_validate did not catch faulty input '{gridsize}', '{walls}', '{entrance}'")

This isn't completely correct though: for 1) I have to restrict myself to strings and 2) this only tests situations where all three of the inputs are wrong, not just one of them.

Hypothesis library: strategy for the complement of some other strategy

2 Answers2