TestNG reports retried methods as separate test runs

Question

I started working with TESTNG for the first time few days back.

I implemented a retry analyzer using the IRetryAnalyzer interface which retries a failed test method for a maximum of 3 times.

I have a test method called retryInvoker() which fails twice and succeeds the third time.

The way TestNG reports this is 3 separate test runs, 2 of which failed and 1 succeeded, which is correct actually. Although I just wish to capture it as a single test run, which ultimately succeeded (if it did before the maximum allowed retries ended, which in this case were 3). Or even if it didn't succeed after 3 retries, I wish to report it as a single test run that failed instead of 4 separate test runs, all of which failed.

Any leads on this?

Just to add detail, I also implemented a listener which implements the IMethodListener interface and sets the retry analyzer for the test class if it doesn't have it already before the test method is invoked. I read about the ITestListener interface that has methods such as the onFailure() method which may be used and also there could be a logic implemented in the afterInvocation() method of the IMethodListener. — laksh, May 16 '18 at 08:19

score 0 · Answer 1 · answered May 17 '18 at 08:42

You could try the approach, suggested in this SO answer, removing the duplicates from the test context in case the same method is found either in failed or passed tests:

@Overrride
public void onFinish(ITestContext context) {
    Iterator<ITestResult> failedTestCases = context.getFailedTests().getAllResults().iterator();
    while (failedTestCases.hasNext()) {
        System.out.println("failedTestCases");
        ITestResult failedTestCase = failedTestCases.next();
        ITestNGMethod method = failedTestCase.getMethod();
        if (context.getFailedTests().getResults(method).size() > 1) {
            System.out.println("failed test case remove as dup:" + failedTestCase.getTestClass().toString());
            failedTestCases.remove();
        } else {
            if (context.getPassedTests().getResults(method).size() > 0) {
                System.out.println(
                        "failed test case remove as pass retry:" + failedTestCase.getTestClass().toString());
                failedTestCases.remove();
            }
        }
    }
}

It surely helps modifying the report for each suite. Although I feel this approach wouldn't work in case we use data providers. — laksh, May 21 '18 at 17:44

TestNG reports retried methods as separate test runs

1 Answers1