Jqwik: How to check if all possibilities is covered?

Question

How to check if all possibilities (cartesian product of arguments) is covered in summary by N properties? Some of them can be tested few times by different properties.

Can you show a concrete example of what you mean? Using exhaustive generation will ensure the Cartesian product of all possible parameter values, but I don’t understand what you mean by „in summary by N properties“. — johanneslink, Mar 17 '22 at 18:40
For example I have a method that returns enum with 10+ values. This function takes 3 arguments with maximum 2x4x3 = 24 cases. One property works for 1x4x3 = 12 cases, second 2x2x3 = 12, third one 1x3x3 = 9. Generally I have tested more cases than cartesian product (24 vs 33) but I want to bu sure that every case is tested at least one time. — Marek Łukasz Urbański, Mar 18 '22 at 11:00

score 0 · Answer 1 · answered Mar 18 '22 at 13:05

There's no way in jqwik to assert exhaustive parameter generation across property methods. What you can do, though, is to check all property conditions in a single property method:

@Property(generation = GenerationMode.EXHAUSTIVE)
void fullCartesianProductProperty(
    @ForAll @IntRange(min = 0, max = 1) int p1,
    @ForAll @IntRange(min = 0, max = 3) int p2,
    @ForAll @IntRange(min = 1, max = 3) int p3
) {
    MyEnum result = toCheck(p1, p2, p3);
    if (p1 > 0) {
        // Will only be executed in 12 cases
        assertThat(result)...;
    }
    if (p2 <= 2) {
        // Will only be executed in 18 cases
        assertThat(result)...;
    }
    if (p3 > 1) {
        // Will only be executed in 16 cases
        assertThat(result)...;
    }
    // Will be executed in all 24 cases
    assertThat(result);
}

In this case, all 24 combinations will be executed.

Mind that generation = GenerationMode.EXHAUSTIVE is only necessary if the size of the cartesian product exceeds a property's number of tries (1000 by default).

Thanks for tip with Exhaustive. It works for my beacuse I have less than 1000 actually. And ForAll works well. This way with if is hard to read. I want to have few properties. I am looking for some assert that I can execute in AfterClass and It will check all cases coverage. I was trying with Statistics methods, but after every property Statistic is cleaned. — Marek Łukasz Urbański, Mar 18 '22 at 13:49

score 0 · Answer 2 · answered Mar 21 '22 at 15:16

I found a solution with AfterContainer, but I have a problem with disappearing Statistics after every Property.

private static final StatisticsCollectorImpl CLASS_STATISTICS_COLLECTOR = new StatisticsCollectorImpl("allCases");

Actually I am using manually created StatisticsCollectorImpl and updating it additionally in every Property. So, in result, I have statistics per property and per class.

Statistics.collect(p1, p2, p3);
CLASS_STATISTICS_COLLECTOR.collect(p1, p2, p3);

Fake property allCasesGenerator generates cartesian product.

@Property(generation = GenerationMode.EXHAUSTIVE)
void allCasesGenerator (
    @ForAll @IntRange(min = 0, max = 1) int p1,
    @ForAll @IntRange(min = 0, max = 3) int p2,
    @ForAll @IntRange(min = 1, max = 3) int p3
) {
    CLASS_STATISTICS_COLLECTOR.collect(p1, p2, p3);
}

In AfterContainer I am checking cases existing only once - only from allCasesGenerator.

@AfterContainer
    static void afterAll() {
        final List<String> presentOnlyInAllCasesGeneratorProperty = CLASS_STATISTICS_COLLECTOR.statisticsEntries()
                .stream()
                .filter(entry -> entry.count() <= 1)
                .map(StatisticsEntryImpl::name)
                .collect(Collectors.toList());
        assertThat(presentOnlyInAllCasesGeneratorProperty)
                .isEmpty();
    }

Jqwik: How to check if all possibilities is covered?

2 Answers2