TestNG&Allure: stop test execution after 1st failure

Question

I have a flaky test which fails once per 10-20 attempts because of intermittently reproducible bug. I'd like to have this test to be marked as Failed after first failure. No further retries needed.

This is how the Test annotation looks like:

 @Test(invocationCount = 20, threadPoolSize = 3)

The problem is that if it fails not in the last round - Allure report treat it as "Flaky" test and the report is green. What I'm trying to achieve is terminate test method retrying after the first failure. This test should be red in Allure report.

score 1 · Answer 1 · answered Dec 23 '19 at 11:32

The annotations are set before runtime, so once you are running the test, you cannot change the invocationCount.

What you can do, is to use a listener to stop the execution when it reaches onTestFailure.

So let's say you have the test class:

import org.testng.ITestContext;
import org.testng.annotations.Test;
import org.testng.asserts.Assertion;
import org.testng.annotations.Listeners;

@Listeners(TestListener.class)
public class CreateBookingTest extends TestConfig {

    @Test(invocationCount = 20)
    public void stopInvocationsTest(ITestContext testContext) {
        int currentCount = testContext.getAllTestMethods()[0].getCurrentInvocationCount();

        // We will force a failure in the invocation 5 (the 6th run).
        if (currentCount==5) {
            new Assertion().fail("Method failed in invocation " + currentCount);
        }
    }

}

Then the listener will abort th execution if there's a single failure and stop from running the rest of the tests:

import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;
import org.testng.asserts.Assertion;

public class TestListener implements ITestListener {

    @Override
    public void onTestStart(ITestResult result) {
        System.out.println("\nSTARTING: " + result.getMethod().getCurrentInvocationCount());
    }

    @Override
    public void onTestSuccess(ITestResult result) {
        System.out.println("PASSED: " + result.getMethod().getCurrentInvocationCount());
    }

    @Override
    public void onTestFailure(ITestResult result) {
        System.out.println("FAILED: " + result.getMethod().getCurrentInvocationCount());
        new Assertion().fail("Aborting");
    }

    @Override
    public void onTestSkipped(ITestResult result) {
    }

    @Override
    public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
    }

    @Override
    public void onStart(ITestContext context) {
    }

    @Override
    public void onFinish(ITestContext context) {
    }

}

TestNG&Allure: stop test execution after 1st failure

1 Answers1