How do the [Retry] and [Repeat] attributes interact, in NUnit

Question

[Retry] will re-run the test if it fails, and only fail if ALL runs fail.
[Repeat] will re-run the test if it passes, and only pass if ALL runs pass.

So ... what's the expected behaviour if you attach both attributes?

Which attribute is on the inside, and which is on the outside?

If [Retry] is on the outside, then you're asserting:

"This has to be generally very stable, but the occasional flicker is OK."
e.g. [Repeat(50), Retry(3)] says "you have to be able to manage 50 successes in a row, but you get 3 attempts to do that."

If [Repeat] is on the outside, then you're asserting:

"This generally doesn't fail ALL the time. It will consistently work if you try a few times."
e.g. [Repeat(50), Retry(4)] says "In pretty much EVERY set of 4, at least one of the executions will have succeeded."

As far as I can tell, the docs I can see don't give any indication. :(

score 1 · Accepted Answer · answered Dec 14 '20 at 19:47

Earlier releases didn't handle this and the result was non-deterministic. A PR in 2018 introduced code, which caused any test with multiple repeating attributes (like Repeat and Retry) to always fail.

In fact, there's even a test in the codebase to verify this behavior: https://github.com/nunit/nunit/blob/aa74dcd07da92e6b3a1182c760b6ca3e72455379/src/NUnitFramework/tests/Attributes/RepeatableTestsWithTimeoutAttributesTests.cs#L128

So unfortunately the answer is that you can't use both attributes on the same test in current releases of NUnit.

Workaround...

Use the attribute you want to be "on the outside" and implement the equivalent of the other attribute in your test code itself.

If you want to test "three chances to get 50 in a row right," use [Retry(3)] and add code in your test to try the operation 50 times, failing as soon as it fails.

for (int i = 0; i < 50; i++)
{
    // Try your operation
    Assert.That(.... // whatever you are asserting
}

If you want to test "run 50 times with 4 retries each time," use [Repeat(50)] and include a loop that implements retry. This one is trickier, since you have to prevent any assertions from terminating the test prematurely. The following is at least a start on what you may need to do.

Exception ex = null;

for (int i = 0; i < 4; i++)
{
    ex = Assert.Catch(() => // try your operation );
    if (ex == null) Assert.Pass();
}

Assert.Fail(ex.Message);

Important: Do not use c# try and catch instead of Assert.Catch. In the latest versions of NUnit, the failure is reported before any exception is thrown. The NUnit exceptions are not intended to be seen by the test code.

How do the [Retry] and [Repeat] attributes interact, in NUnit

1 Answers1