Handling of pydantic ValidationError when testing with hypothesis.given

Question

When using hypothesis to test my pydantic models, I do not know how to handle ValidationError thrown by custom validators. This is a very small example that shows the problem:

# model
from pydantic import BaseModel, validator

class SimpleModel(BaseModel):
    a: int
    b: int

    @validator('b')
    def check_numbers(cls, b, values):
        if b*values['a'] < 0:
            raise ValueError('a*b > 0 does not hold')
        return b

# test
from hypothesis import given, strategies as st

@given(st.builds(SimpleModel))
def test_simple_model(instance: SimpleModel):
    assert type(instance.b) == int

Up to now I have written custom hypothesis search strategies to only generate instances that are valid. But this gets very tedious for more complex models, so in my opinion there has to be a smarter way to "use" the ValidationError. The error is also raised before the test function, therefore I cannot handle it in the test function.

I would need a possibility to generate instances, that simply skips instances that raise a ValidationError.

Simon · Accepted Answer · 2023-05-09T12:43:05.627

After using more of the functionality of hypothesis, I came up with this approach, which uses composite strategy and assume. With composite I create a custom strategy to be used in the given decorator. Inside assume is used to tell hypothesis that examples are bad and should be skipped whenever a ValidationError is thrown.

from pydantic import ValidationError
from hypothesis import given, assume, strategies as st

@st.composite
def simple_model_strategy(draw) -> SimpleModel:
    try:
        simple_model = draw(st.builds(SimpleModel))
    except ValidationError:
        assume(False)
    return simple_model

@given(simple_model_strategy())
def test_simple_model(instance: SimpleModel):
    assert type(instance.b) == int

Notes:

I used assume in the test before, but like this it can also be used during model generation.
If assume is called on most of the autogenerated instances, hypothesis warns about problems with the test case.

Handling of pydantic ValidationError when testing with hypothesis.given

1 Answers1