How to send TestNG DefaultSuite results after retryAnalyzer to Maven Surefire Plugin

Question

I did search a lot and this is the closest answer that I got but does not solve my problem. TestNG retrying failed tests doesn't output the correct test results

But I need the above problem to extend from TestNG onto Maven. Please help.

My Project stack: TestNG, Maven surefire plugin, Maven. I am running the tests from command line - "mvn clean compile test". I am using a retryanalyzer to re-run the failed tests, that are passing the second time. I have been able to use a CustomListener to update the results stored in the TestNG's testcontext (based on solutions in stackoverflow). This was confirmed when I ran the tests as TestNG suite.

But when I run "mvn clean compile test", all the retries are counted as separate tests and build always fails. How do I make TestNG send only the final Suite results to Maven Surefire plugin ?

TESTNG EXECUTION:

============================================
DefaultTest
Total tests run: 4, Failures: 2, Skips: 0
============================================

============================================
DefaultSuite
Total tests run: 2, Failures: 0, Skips: 0
============================================

MAVEN EXECUTION:

Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.292 sec <<< FAILURE!

Results :

Failed tests: 
  test1(foo.TestClass1)
  test1(foo.TestClass1)

Tests run: 4, Failures: 2, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

The build passed because of the retry. How do I reflect the TestNG retry Suite results onto Maven ?

Hi! Have you resolved your issue? I tracked the issue down to maven testng runner which installs its own listener into testng. It disregards final counter adjustments since it maintains its own counters — WarGoth, Feb 20 '15 at 19:26

Morvader · Answer 1 · 2015-03-18T15:20:32.043

Finally got it. I use this code:

ListenerApadter:

public class MyTestListenerAdapter extends TestListenerAdapter {
    @Override
    public void onTestFailure(ITestResult result) {
        if (result.getMethod().getRetryAnalyzer() != null) {
            MyRetryAnalyzer retryAnalyzer = (MyRetryAnalyzer)result.getMethod().getRetryAnalyzer();

            if(retryAnalyzer.isRetryAvailable()) {
                result.setStatus(ITestResult.SKIP);
            } else {
                result.setStatus(ITestResult.FAILURE);
            }
            Reporter.setCurrentTestResult(result);
        }
    }

   @Overrride
   public void onFinish(ITestContext context) {
     Iterator<ITestResult> failedTestCases =context.getFailedTests().getAllResults().iterator();
    while (failedTestCases.hasNext()) {
        System.out.println("failedTestCases");
        ITestResult failedTestCase = failedTestCases.next();
        ITestNGMethod method = failedTestCase.getMethod();
        if (context.getFailedTests().getResults(method).size() > 1) {
            System.out.println("failed test case remove as dup:" + failedTestCase.getTestClass().toString());
            failedTestCases.remove();
        } else {

            if (context.getPassedTests().getResults(method).size() > 0) {
                System.out.println("failed test case remove as pass retry:" + failedTestCase.getTestClass().toString());
                failedTestCases.remove();
            }
        }
    }
   }
}

RetryAnalizer:

public class MyRetryAnalyzer implements IRetryAnalyzer {
    private static int MAX_RETRY_COUNT = 3;

    AtomicInteger count = new AtomicInteger(MAX_RETRY_COUNT);

    public boolean isRetryAvailable() {
        return (count.intValue() > 0);
    }

    @Override
    public boolean retry(ITestResult result) {
        boolean retry = false;
        if (isRetryAvailable()) {
            System.out.println("Going to retry test case: " + result.getMethod() + ", " + (MAX_RETRY_COUNT - count.intValue() + 1) + " out of " + MAX_RETRY_COUNT);
            retry = true;
            count.decrementAndGet();
        }
        return retry;
    }
}

POM.xml -> Surefire Configuration:

This is where you should configure "overwrite" surefire listener wich has their own counters.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.18.1</version>
  <configuration>
    <suiteXmlFiles><suiteXmlFile>${basedir}/testng.xml</suiteXmlFile></suiteXmlFiles>
 <properties> 
   <property>
    <name>listener</name>
    <value>Utils.MyTestListenerAdapter,Utils.MyRetryAnalizer</value>
   </property>
 </properties>

I have a similar setup in my test Framework but I get the following error: Listener package.RerunTestAnalyzer@6fda4e65 must be one of ITestListener, ISuiteListener, IReporter, IAnnotationTransformer, IMethodInterceptor or IInvokedMethodListener. Do you know how I can fix this error? — Ziska, Sep 30 '15 at 03:28
After some research, I found that we do not need to declare our custom implementation of IRetryAnalyzer in pom.xml. As it is not a typical TestNG listener, the error above is valid. Besides, the logic that corrects the total number of executed tests is in TestListenerAdapter implementation. As long as we declare this listener in pom file, Maven will print the correct number of tests executed. — Ziska, Sep 30 '15 at 04:23

score 0 · Answer 2 · answered Mar 29 '15 at 10:50

Some suit info is incorrect, but maven surefire works properly

maven-surefire-plugin 2.16
testng 6.8.8

// ===== InvokedMethodListener =====
import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;

public class InvokedMethodListener implements IInvokedMethodListener {

    @Override
    public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
    }

    @Override
    public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
        RetryAnalyzer retryAnalyzer = (RetryAnalyzer)testResult.getMethod().getRetryAnalyzer();
        if (retryAnalyzer == null || retryAnalyzer.isFailed()) {
            return;
        }
        if (testResult.getStatus() == ITestResult.FAILURE) {
            testResult.setStatus(ITestResult.SUCCESS);
        }
    }

}

// ===== RetryAnalyzer =====
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
import org.testng.Reporter;

public class RetryAnalyzer implements IRetryAnalyzer {
    private int count = 0;
    private int maxCount = 2;

    @Override
    public boolean retry(ITestResult result) {
        if (!result.isSuccess()) {
            if (count < maxCount) {
                count++;
                return true;
            }
        }
        return false;
    }

    public boolean isFailed() {
        return count >= maxCount;
    }
}


// ===== BaseTest =====
@Listeners({ ..InvokedMethodListener.class })
public abstract class BaseTest {
...
    @BeforeSuite(alwaysRun = true)
    public void beforeSuite(ITestContext context) {
        for (ITestNGMethod method : context.getAllTestMethods()) {
            method.setRetryAnalyzer(new RetryAnalyzer());
        }
    }
}

How to send TestNG DefaultSuite results after retryAnalyzer to Maven Surefire Plugin

2 Answers2