TestNG retrying failed tests doesn't output the correct test results

Question

Setup: I have a class that extends the IRetryAnalyzer and have implemented a simple retry logic overriding the following method: public boolean retry(ITestResult result) {

I have another class that extends the class TestListenerAdapter that retries tests that failed until they pass or report failures. I have implemented my logic overriding the following method: public void onTestFailure(ITestResult result) {

Scenario: I have a total of 10 tests. 1 out of the 10 tests fail 2 times and pass on the 3rd attempt with my retry logic. The test results show the following: Total tests: 12, Failed: 2, Skipped: 0

What i would like is to output the correct number of tests run. And also disregard the 2 failures since the test passed at the end. So the result should look something like this: Total tests: 10, Failed:0, Skipped: 0

What am i missing here? Do i need to modify the ITestResult object? If yes, how?

FYI: I was able to achieve this using JUnit (implementing the TestRule interface).

Thanks in advance.

Answer 1

Please consider the following test results with max. 2 Retries:

1. Passed => Overall ok!
2. Failed, Passed => Overall ok!
3. Failed, Failed, Passed => Overall ok!
4. Failed, Failed, Failed => Overall Fail!

What i did is to create a TestNg listener which extends the default behaviour and does some magic after all tests are finished.

public class FixRetryListener extends TestListenerAdapter {

    @Override
    public void onFinish(ITestContext testContext) {
        super.onFinish(testContext);

        // List of test results which we will delete later
        List<ITestResult> testsToBeRemoved = new ArrayList<>();

        // collect all id's from passed test
        Set <Integer> passedTestIds = new HashSet<>();
        for (ITestResult passedTest : testContext.getPassedTests().getAllResults()) {
            passedTestIds.add(TestUtil.getId(passedTest));
        }

        Set <Integer> failedTestIds = new HashSet<>();
        for (ITestResult failedTest : testContext.getFailedTests().getAllResults()) {

            // id = class + method + dataprovider
            int failedTestId = TestUtil.getId(failedTest);

            // if we saw this test as a failed test before we mark as to be deleted
            // or delete this failed test if there is at least one passed version
            if (failedTestIds.contains(failedTestId) || passedTestIds.contains(failedTestId)) {
                testsToBeRemoved.add(failedTest);
            } else {
                failedTestIds.add(failedTestId);
            }
        }

        // finally delete all tests that are marked
        for (Iterator<ITestResult> iterator = testContext.getFailedTests().getAllResults().iterator(); iterator.hasNext(); ) {
            ITestResult testResult = iterator.next();
            if (testsToBeRemoved.contains(testResult)) {
                iterator.remove();
            }
        }

    }

}

Basically i do 2 things:

1. Collect all passed test. If i encounter a failed test with at least one passed test i remove the failed test (That would cover case 2 and 3 from above)
2. Iterate over all failed test. If i encounter a failed test which previously failed i remove the current failed result. (That would cover case 3 and 4 actually). That also means i will only keep the first failed result if there are several failed results.

To identify a testresult i use the following simple hash function:

public class TestUtil {

    public static int getId(ITestResult result) {
        int id = result.getTestClass().getName().hashCode();
        id = 31 * id + result.getMethod().getMethodName().hashCode();
        id = 31 * id + (result.getParameters() != null ? Arrays.hashCode(result.getParameters()) : 0);
        return id;
    }
}

This approach does work with dataproviders as well but has one small limitation! If you use dataproviders with random values you'll run into problems:

@DataProvider(name = "dataprovider")
public Object[][] getData() {
    return new Object[][]{{System.currentTimeMillis()}};
}

For failed tests the dataprovider is reevaluated. Therefore you will get a new timestamp and the result1 and result2 for the same mehod will result in different hash values. Solution would be to include the parameterIndex in the getId() Method instead of parameters but it seems such a value is not included in ITestResult.

See this simple example as proof of concept:

@Listeners(value = FixRetryListener.class)
public class SimpleTest {

    private int count = 0;

    @DataProvider(name = "dataprovider")
    public Object[][] getData() {
        return new Object[][]{{"Run1"},{"Run2"}};
    }

    @Test(retryAnalyzer = RetryAnalyzer.class, dataProvider = "dataprovider")
    public void teste(String testName) {
        count++;
        System.out.println("---------------------------------------");
        System.out.println(testName + " " + count);
        if (count % 3 != 0) {
            Assert.fail();
        }

        count = 0;
    }

}

Will yield in:

Total tests run: 2, Failures: 0, Skips: 0

Answer 2

I tried, tried and tried. But now finally it worked

MyRetryAnalyzer.java

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

import java.util.concurrent.atomic.AtomicInteger;


public class MyRetryAnalyzer implements IRetryAnalyzer {
private static int MAX_RETRY_COUNT = 3;

AtomicInteger count = new AtomicInteger(MAX_RETRY_COUNT);

public boolean isRetryAvailable() {
    return (count.intValue() > 0);
}

@Override
public boolean retry(ITestResult result) {
    boolean retry = false;
    if (isRetryAvailable()) {
        System.out.println("Going to retry test case: " + result.getMethod() + ", " + (MAX_RETRY_COUNT - count.intValue() + 1) + " out of " + MAX_RETRY_COUNT);
        retry = true;
        count.decrementAndGet();
    }
    return retry;
}
}

MyTestListenerAdapter.java

import org.testng.*;

import java.util.*;


public class MyTestListenerAdapter extends TestListenerAdapter {
    @Override
    public void onTestFailure(ITestResult result) {
        if (result.getMethod().getRetryAnalyzer() != null) {
            MyRetryAnalyzer retryAnalyzer = (MyRetryAnalyzer)result.getMethod().getRetryAnalyzer();

            if(retryAnalyzer.isRetryAvailable()) {
            } else {
                result.setStatus(ITestResult.FAILURE);
            }
            Reporter.setCurrentTestResult(result);
        }
    }

@Override
    public void onFinish(ITestContext context) {
        Iterator<ITestResult> failedTestCases =context.getFailedTests().getAllResults().iterator();
        while (failedTestCases.hasNext()) {
            System.out.println("failedTestCases");
            ITestResult failedTestCase = failedTestCases.next();
            ITestNGMethod method = failedTestCase.getMethod();
            if (context.getFailedTests().getResults(method).size() > 1) {
                System.out.println("failed test case remove as dup:" + failedTestCase.getTestClass().toString());
                failedTestCases.remove();
            } else {

                if (context.getPassedTests().getResults(method).size() > 0) {
                    System.out.println("failed test case remove as pass retry:" + failedTestCase.getTestClass().toString());
                    failedTestCases.remove();
                }
            }
        }
    }
}

You test class

@Listeners(value = MyTestListenerAdapter.class)

public class Test  {

//Your data provider
@DataProvider

@Test(retryAnalyzer = MyRetryAnalyzer.class)
public void testMethod () {
    //your code goes here
 }

}

Answer 3

My approach builds on Morvader's answer but adds the ability to define retry analyzers that adhere to the original intent of actually failing the test even if the method have passed after some retries.

I also didn't find a need to tend to the number of test in the onFinish() method, the numbers seemed fine in maven-surefire-plugin version 2.18

RetryListenerAdapter

public class RetryListenerAdapter extends TestListenerAdapter {

    @Override
    public void onTestFailure(ITestResult tr) {
        IRetryAnalyzer retryAnalyzer = tr.getMethod().getRetryAnalyzer();
        if (retryAnalyzer == null || !(retryAnalyzer instanceof IRetryAnalyzerWithSkip)) {
            super.onTestFailure(tr);
        } else if (((IRetryAnalyzerWithSkip) retryAnalyzer).isRetryable()) {
            tr.setStatus(ITestResult.SKIP);
            super.onTestSkipped(tr);
        } else {
            super.onTestFailure(tr);
        }
    }
}

IRetryAnalyzerWithSkip

public interface IRetryAnalyzerWithSkip extends IRetryAnalyzer {
    boolean isRetryable();
}

Retry

public class Retry implements IRetryAnalyzerWithSkip {
    private int retryCount = 0;
    private int maxRetryCount = 3;

    public boolean retry(ITestResult result) {

        if (retryCount < maxRetryCount) {
            retryCount++;
            return true;
        }
        return false;
    }

    @Override
    public boolean isRetryable() {
        return retryCount < maxRetryCount;
    }
}

Answer 4

I use this approach:

ListenerApadter:

public class MyTestListenerAdapter extends TestListenerAdapter {
    @Override
    public void onTestFailure(ITestResult result) {
        if (result.getMethod().getRetryAnalyzer() != null) {
            MyRetryAnalyzer retryAnalyzer = (MyRetryAnalyzer)result.getMethod().getRetryAnalyzer();

            if(retryAnalyzer.isRetryAvailable()) {
                result.setStatus(ITestResult.SKIP);
            } else {
                result.setStatus(ITestResult.FAILURE);
            }
            Reporter.setCurrentTestResult(result);
        }
    }

   @Overrride
   public void onFinish(ITestContext context) {
     Iterator<ITestResult> failedTestCases =context.getFailedTests().getAllResults().iterator();
    while (failedTestCases.hasNext()) {
        System.out.println("failedTestCases");
        ITestResult failedTestCase = failedTestCases.next();
        ITestNGMethod method = failedTestCase.getMethod();
        if (context.getFailedTests().getResults(method).size() > 1) {
            System.out.println("failed test case remove as dup:" + failedTestCase.getTestClass().toString());
            failedTestCases.remove();
        } else {

            if (context.getPassedTests().getResults(method).size() > 0) {
                System.out.println("failed test case remove as pass retry:" + failedTestCase.getTestClass().toString());
                failedTestCases.remove();
            }
        }
    }
   }
}

RetryAnalizer:

public class MyRetryAnalyzer implements IRetryAnalyzer {
    private static int MAX_RETRY_COUNT = 3;

    AtomicInteger count = new AtomicInteger(MAX_RETRY_COUNT);

    public boolean isRetryAvailable() {
        return (count.intValue() > 0);
    }

    @Override
    public boolean retry(ITestResult result) {
        boolean retry = false;
        if (isRetryAvailable()) {
            System.out.println("Going to retry test case: " + result.getMethod() + ", " + (MAX_RETRY_COUNT - count.intValue() + 1) + " out of " + MAX_RETRY_COUNT);
            retry = true;
            count.decrementAndGet();
        }
        return retry;
    }
}

POM.xml -> Surefire Configuration:

This is where you should configure "overwrite" surefire listener wich has their own counters.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.18.1</version>
  <configuration>
    <suiteXmlFiles><suiteXmlFile>${basedir}/testng.xml</suiteXmlFile></suiteXmlFiles>
 <properties> 
   <property>
    <name>listener</name>
    <value>Utils.MyTestListenerAdapter,Utils.MyRetryAnalizer</value>
   </property>
 </properties>

Answer 5

This is fixed without having to do any extra magic in your Retry Analyzer in the latest version of TestNG 7.1.0. Tests that were retried will now be labeled as "retried" and you can see this in the emailable-report.html report that gets generated after runs. If you'd still like to do some logic with the tests as they are being retried you can do this:

@AfterMethod(alwaysRun = true)
    public void afterMethod( ITestResult result) throws Exception {
            if (result.wasRetried()){
                //your logic here
            }