Assertions for non-deterministic behavior

Question

Does AssertJ (or JUnit) have a way to chain, in a single (fluent) expression, several assertions on the same unit under test where one of the assertions may throw an exception. Essentially, I'm trying to assert that:

If a unit under test (X) doesn't result in a particular exception, which it may, then assert that a particular property on the unit under test doesn't hold. Otherwise assert the exception is of a certain type.

For example, is there a way to express the assertion that the following erroneous code could EITHER result in an Exception or in a situation where strings.size() != 10000:

@Test/*(expected=ArrayIndexOutOfBoundsException.class)*/
public void raceConditions() throws Exception {


    List<String> strings = new ArrayList<>(); //not thread-safe

    Stream.iterate("+", s -> s+"+")
    .parallel()
    .limit(10000)
    //.peek(e -> System.out.println(e+" is processed by "+ Thread.currentThread().getName()))
    .forEach(e -> strings.add(e));

    System.out.println("# of elems: "+strings.size());
}

AssertJ has a concept of soft assertions, are those to be used in the scenarios like that? I'd appreciate some code samples if so.

Or perhaps there are better frameworks specifically design for this type of scenarios?

Thanks.

I think there's two things going on here at the same time. You're trying to force a race condition, and you're trying to test something that only happens if an exception occurs, which it might not. Unit tests should be deterministic. If you're writing a non-deterministic one then you're not writing a unit test, but something else. Checking for race conditions is important, but usually that's a stress test, or a fuzzing test, not a unit test. A unit test should produce consistent, predictable results. A stress test produces statistical data. — tadman, Feb 13 '18 at 19:24
may be this would be an option? `if(yourConditionIsNotMet) {Assert.fail(SomeCustomException)}`, this `SomeCustomException` would denote the race — Eugene, Feb 13 '18 at 19:25
@tadman they are deterministic in a sense that I'm expecting a consistent and predictable failure in both cases. What's open is a type of failure. Is there anything that allows to assert presence of a race condition? — Simeon Leyzerzon, Feb 13 '18 at 19:30
I don't think you should be testing for race conditions. You should be testing that your operations perform as expected under normal conditions. A secondary stress-testing framework can shake out race conditions if that's a concern. That's where you can abuse the system to the point of failure so you can understand all the ways in which your code can explode. Some of those failures might be bugs, and some of those bugs might need unit tests that verify they've been fixed. — tadman, Feb 13 '18 at 19:32
Well, my goal is not `to test` per say, as I already know this code produces errors. What I want to do is express (via assertions) that this behavior is erroneous. The reason I mention testing frameworks is because they allow to model assertions quite naturally. Perhaps there are other means which I'm not familiar with, I'm certainly open to suggestions. Thanks. — Simeon Leyzerzon, Feb 13 '18 at 19:41
This doesn’t make any sense. Non-deterministic means non-deterministic. There is no guaranty that this code exhibits either of these two behaviors. It’s perfectly possible that the resulting list’s `size()` method returns `10000`, which doesn’t imply that the list is in a valid, consistent state. It may contain wrongly ordered elements, spurious `null`s and duplicates or even an array shorter than the reported size. But even having an entirely correct list result is among the possible results. It might be that the code never completes. That’s *non-deterministic*. — Holger, Feb 14 '18 at 08:12

score 1 · Accepted Answer · answered Feb 14 '18 at 07:05

I'm not sure if that is what you are really looking for but you can try using assumptions. After executing the code under test, perform an assumption on the result, the following code/assertions will only be executed if the assumptions were correct.

Since 3.9.0 AssertJ provides assumptions out of the box, example:

List<String> strings = new ArrayList<>(); // not thread-safe

int requestedSize = 10_000;

Throwable thrown = catchThrowable(() -> {
  Stream.iterate("+", s -> s + "+")
        .parallel()
        .limit(requestedSize)
        .forEach(strings::add);
});

// thrown is null if there was no exception raised
assumeThat(thrown).isNotNull();

// only executed if thrown was not null otherwise the test is skipped.
assertThat(thrown).isInstanceOf(ArrayIndexOutOfBoundsException.class);

You should also have a look at https://github.com/awaitility/awaitility if you are testing asynchronous code.

Simeon Leyzerzon · Answer 2 · 2018-02-13T21:26:08.830

I came up with the following:

 @Test
 public void testRaceConditions() {
    List<String> strings = new ArrayList<>(); //not thread-safe

    int requestedSize = 10_000;

    Throwable thrown = catchThrowable(() -> { 
    Stream.iterate("+", s -> s+"+")
    .parallel()
    .limit(requestedSize)
    //.peek(e -> System.out.println(e+" is processed by "+ Thread.currentThread().getName()))
    .forEach(e -> strings.add(e));
    });

    SoftAssertions.assertSoftly(softly -> {
         softly.assertThat(strings.size()).isNotEqualTo(requestedSize);
         softly.assertThat(thrown).isInstanceOf(ArrayIndexOutOfBoundsException.class);
     });

 }

If somebody with more of AssertJ under their belt or some other tools know of a better way, I'd gladly accept their solutions. Thanks!

Assertions for non-deterministic behavior

2 Answers2