Retry failed test case in testng

Question

I am working on a testng project and my goal is that let testng auto retry the failed test cases. For example, there are 10 test cases in the first round and 5 failed. So after first round, I let testng select the 5 failed test cases and rerun them again. In the second round, maybe there are 2 failed test cases, then I rerun this 2 agian.

I have tried the IRetryAnalyzer but it is different. The IRetryAnalyzer is retrying the failed test cases immediatelly instead of the end of each round.

So currently I want to call the retry using onStart and onFinish in the ISuiteListener. In this case I implement the onFinish method like this:

@Override
public void onFinish(ISuite suite) {
    logger.info("Round " + retryCounter
            + " Testing suit stops. onFinish method is invoked.");
    if (!doRetry()) {
        logger.info("Retry finished.");
        cleanReport(suite);
        return;
    }

    // I want to remove the passed cases here
    // and create a suite to run the failed test cases.
    suite.run();
}

So is it possible do that? Or any better idea for this requirement.

Answer 1

It looks like the question is old, but still unanswered, so here is my solution.

Firstly, you have to implement IRetryAnalyzer interface like this:

    public class Retry implements IRetryAnalyzer {

    private int retryCount = 0;
    private int maxRetryCount = 1;

    public boolean retry(ITestResult result) {
        if (retryCount < maxRetryCount) {
            System.out.println("Retrying test " + result.getName() + " with status "
                    + getResultStatusName(result.getStatus()) + " for the " + (retryCount+1) + " time(s).");
            retryCount++;
            return true;
        }
        return false;
    }

    public String getResultStatusName(int status) {
        String resultName = null;
        if(status==1)
            resultName = "SUCCESS";
        if(status==2)
            resultName = "FAILURE";
        if(status==3)
            resultName = "SKIP";
        return resultName;
    }
}

Here, maxRetryCount is the number of times, you want your failed tests to be re-run. Tests are re-run immediately after their fail.

Secondly, implement IAnnotationTransformer interface:

public class RetryListener implements IAnnotationTransformer {

    @Override
    public void transform(ITestAnnotation testannotation, Class testClass,
                          Constructor testConstructor, Method testMethod)   {
        IRetryAnalyzer retry = testannotation.getRetryAnalyzer();

        if (retry == null)  {
            testannotation.setRetryAnalyzer(Retry.class);
        }

    }
}

Now, in your TestNG xml suite with tests, you should add listener:

<listeners>
    <listener class-name="Utils.reRunTest.RetryListener"/>
</listeners>

You can also add it to your pom.xml file:

 <properties>
     <property>
         <name>usedefaultlisteners</name>
         <value>false</value>
     </property>
     <property>
         <name>listener</name>
         <value>org.testng.reporters.EmailableReporter2, org.testng.reporters.XMLReporter, org.testng.reporters.FailedReporter, Utils.reRunTest.RetryListener
         </value>
     </property>

You may also want your test re-runs not to affect total number of tests. This is done, by implementing TestListener interface and adding it as a listener alongside with RetryAnalyzer. The implementation that will remove flaky tests and failed tests from total number of tests is:

public class TestListener implements ITestListener {
    @Override
    public void onFinish(ITestContext context) {
        Set<ITestResult> failedTests = context.getFailedTests().getAllResults();
        Set<ITestResult> skippedTests = context.getSkippedTests().getAllResults();
        for (ITestResult temp : failedTests) {
            ITestNGMethod method = temp.getMethod();
            if (context.getFailedTests().getResults(method).size() > 1) {
                failedTests.remove(temp);
            } else {
                if (context.getPassedTests().getResults(method).size() > 0) {
                    failedTests.remove(temp);
                }
            }
        }
        for (ITestResult temp : skippedTests) {
            ITestNGMethod method = temp.getMethod();
            if (context.getSkippedTests().getResults(method).size() > 1) {
                skippedTests.remove(temp);
            } else {
                if (context.getPassedTests().getResults(method).size() > 0) {
                    skippedTests.remove(temp);
                }
            }
        }
    }

    public void onTestStart(ITestResult result) {
    }

    public void onTestSuccess(ITestResult result) {
    }

    public void onTestFailure(ITestResult result) {
    }

    public void onTestSkipped(ITestResult result) {
    }

    public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
    }

    public void onStart(ITestContext context) {
    }
}

Hope that helps. P.S Don't use it with TestNG version 6.9.10 `cause it has some issues and you re-run tests will always pass, disregarding their real state.

Answer 2

Testng creates a testng-failed.xml in the output folder after a run, which you can run after running the testng.xml. So you have the 10 testcases in testng.xml and run them and whatever fails will come in the testng-failed.xml which you can run then.

Answer 3

If testng version is 7.4.0 or more then retry test cases are getting total count and it is also present skip section as well.

To solve this problem we need to do a little code change in TestListener class.

The following code snippet worked for me.

public class RetryAnalyzer extends RetryAnalyzerCount
{

    @Override
    public boolean retryMethod(ITestResult result)
    {
        int count = 0;
        int trackingCount = 0;
        Properties prop = new Properties();
        String dir = null;
        FileInputStream input = null;
        
        try
        {
            //Reading Property file to get Retry Count.
            dir = new File(System.getProperty("user.dir")).getParent();
            dir = dir + "Your Properties Path";
            input = new FileInputStream(dir);
            prop.load(input);
        }
        catch (FileNotFoundException e)
        {
            count = 1;
            e.printStackTrace();
        }
        catch (IOException e)
        {
            e.printStackTrace();
        }
        count = Integer.parseInt(prop.getProperty("RetryCount"));
        
        if (!result.isSuccess())
        {
            while (count != 0)
            {
                count--;
                trackingCount++;
                result.setStatus(ITestResult.SUCCESS_PERCENTAGE_FAILURE);
                String message = Thread.currentThread().getName() + "Error in '" + result.getName() + "' with status '" + result.getStatus() + "'. Retrying '" + trackingCount + "' times.";
                Reporter.log(message, true);
                return true;
            }
        }
        return false;
        
    }
}

In this we are maintaining thre retry count in a property file so that we can change the retry count without compiling the code.

Below is values of properties file.

RetryCount=1

After that we need to change TestListener class.

    @Override
    public void onFinish(ITestContext context)
    {
        //extent.flush();
        Set<ITestResult> failedTests = context.getFailedTests().getAllResults();
        Set<ITestResult> skippedTests = context.getSkippedTests().getAllResults();
        Set<ITestResult> passedTests = context.getPassedTests().getAllResults();
        for (ITestResult temp : failedTests) {
            ITestNGMethod method = temp.getMethod();
            System.out.println("Method Name  " + method.getMethodName() + "  Have Result " + temp.getStatus() + "  Have Retries ?  " +temp.wasRetried() + " \n " + " have result " +context.getFailedTests().getResults(method).toString());
            if (temp.wasRetried()){
                failedTests.remove(temp);
            }
        }
        for (ITestResult temp : skippedTests) {
            ITestNGMethod method = temp.getMethod();
            System.out.println("Method Name  " + method.getMethodName() + "  Have Result " + temp.getStatus() + "   Have Retries ?  " +temp.wasRetried() + " \n " + " Have Result " + context.getSkippedTests().getResults(method).toString());
            if (temp.wasRetried()){
                skippedTests.remove(temp);
            }
        }
        for (ITestResult temp : passedTests) {
            ITestNGMethod method = temp.getMethod();
            System.out.println("Method Name  " + method.getMethodName() + "  Have Result " + temp.getStatus() + "   Have Retries ?  " +temp.wasRetried() + " \n " + " Have Result " + context.getPassedTests().getResults(method).toString());
            if (temp.wasRetried()){
                passedTests.remove(temp);
            }
        }
   }

It will remove the retry test cases count from skip section.

Now if you retrying the testcases at testcase level the only need to add

@Test(retryAnalyzer=RetryAnalyzer.class,timeOut = DEFAULT_TEST_TIMEOUT, description = "XXX", dataProvider = "YYY", groups ={"ZZZ"})
    public void test_A() {

    }

But if You are retry testcase at suite level then need to create another class

public class AnnotationTransformer implements IAnnotationTransformer {

    @Override
    public void transform(ITestAnnotation annotation, Class testClass,Constructor testConstructor ,Method testMethod)
    {
        annotation.setRetryAnalyzer(RetryAnalyzer.class);
    }
}

And then call it in testng.xml file.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
    <suite name="RetryFailedTests" verbose="1" preserve-order="true" parallel="methods" thread-count="1" >
  <listeners>
        <listener class-name="AnnotationTransformer"/>
  </listeners>
  <test name="RetryMulitple">
        <classes> 
            <class name="Your TestCase"></class>   
        </classes>
    </test>
 </suite>