selenium testNG retry with incorrect result account

Question

I am using testNG 6.9.10 that installed in Eclipse. I was trying to use retry to make sure the failed tests could run maxcount times that defined. See below codes.

public class TestRetry implements IRetryAnalyzer {
    private int retryCount = 0;
    private int maxRetryCount = 1;

    public boolean retry(ITestResult result) {

        if (retryCount < maxRetryCount) {
            retryCount++;
            return true;
        }
        return false;
    }

    @Test(retryAnalyzer = TestRetry.class)
    public void testGenX() {
        Assert.assertEquals("google", "google");
    }

    @Test(retryAnalyzer = TestRetry.class)
    public void testGenY() {
        Assert.assertEquals("hello", "hallo");

    }
}

I got below result:

===============================================
    Default test
    Tests run: 3, Failures: 1, Skips: 1
===============================================


===============================================
Default suite
Total tests run: 3, Failures: 1, Skips: 1
===============================================

But seems like the result count with some problems. I want below:

===============================================
    Default test
    Tests run: 2, Failures: 1, Skips: 0
===============================================


===============================================
Default suite
Total tests run: 2, Failures: 1, Skips: 0
===============================================

I tried to defined the listeners to implement it, something like to override the onFinish function. You may find it in http://www.seleniumeasy.com/testng-tutorials/retry-listener-failed-tests-count-update But finally not works.

can someone who had met this could help?

Answer 1

Its working fine, i hope there is some problem on listener usage. I created TestRetry as same like you but with out @Test methods.

 public class TestRetry implements IRetryAnalyzer{

private int retryCount = 0;
private int maxRetryCount = 1;

@Override
public boolean retry(ITestResult arg0) {
    // TODO Auto-generated method stub
    if (retryCount < maxRetryCount) {
        retryCount++;
        return true;
    }

    return false;
}
}

Created Listener class

 public class TestListener implements ITestListener{

@Override
public void onFinish(ITestContext context) {
    // TODO Auto-generated method stub
    Set<ITestResult> failedTests = context.getFailedTests().getAllResults();
    for (ITestResult temp : failedTests) {
        ITestNGMethod method = temp.getMethod();
        if (context.getFailedTests().getResults(method).size() > 1) {
            failedTests.remove(temp);
        } else {
            if (context.getPassedTests().getResults(method).size() > 0) {
                failedTests.remove(temp);
            }
        }
    }
}

@Override
public void onStart(ITestContext arg0) {
    // TODO Auto-generated method stub

}

@Override
public void onTestFailedButWithinSuccessPercentage(ITestResult arg0) {
    // TODO Auto-generated method stub

}

@Override
public void onTestFailure(ITestResult arg0) {
    // TODO Auto-generated method stub

}

@Override
public void onTestSkipped(ITestResult arg0) {
    // TODO Auto-generated method stub

}

@Override
public void onTestStart(ITestResult arg0) {
    // TODO Auto-generated method stub

}

@Override
public void onTestSuccess(ITestResult arg0) {
    // TODO Auto-generated method stub

}

}

Finally my test class with those methods

  public class RunTest {


@Test(retryAnalyzer = TestRetry.class)
public void testGenX() {
    Assert.assertEquals("google", "google");
}

@Test(retryAnalyzer = TestRetry.class)
public void testGenY() {
    Assert.assertEquals("hello", "hallo");

}

}

Executed this RunTest from testng.xml file by specifying the my custom listener

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite1" parallel="false" preserve-order="true">
<listeners>
   <listener class-name="com.test.TestListener"/>
</listeners>

<test name="TestA">
<classes>
  <class name="com.test.RunTest"/>
</classes>
 </test> <!-- Test -->
 </suite> <!-- Suite -->

Please have a try..

Thank You, Murali

Answer 2

@murali could please see my codes below? I really cannot see any difference. The CustomLinstener.java

package cases;
import java.util.Set;
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestNGMethod;
import org.testng.ITestResult;

public class CustomLinstener implements ITestListener{

    @Override
    public void onFinish(ITestContext context) {
        Set<ITestResult> failedTests = context.getFailedTests().getAllResults();
        for (ITestResult temp : failedTests) {
            ITestNGMethod method = temp.getMethod();
            if (context.getFailedTests().getResults(method).size() > 1) {
                failedTests.remove(temp);
            } else {
                if (context.getPassedTests().getResults(method).size() > 0) {
                    failedTests.remove(temp);
                }
            }
        }
    }

    @Override
    public void onStart(ITestContext arg0) {
        // TODO Auto-generated method stub

    }

    @Override
    public void onTestFailedButWithinSuccessPercentage(ITestResult arg0) {
        // TODO Auto-generated method stub

    }

    @Override
    public void onTestFailure(ITestResult arg0) {
        // TODO Auto-generated method stub

    }

    @Override
    public void onTestSkipped(ITestResult arg0) {
        // TODO Auto-generated method stub

    }

    @Override
    public void onTestStart(ITestResult arg0) {
        // TODO Auto-generated method stub

    }

    @Override
    public void onTestSuccess(ITestResult arg0) {
        // TODO Auto-generated method stub

    }
}

The RunTest.java

package cases;

import org.testng.Assert;
import org.testng.annotations.Test;

public class RunTest {


@Test(retryAnalyzer = TestRetry.class)
public void testGenX() {
    Assert.assertEquals("google", "google");
}

@Test(retryAnalyzer = TestRetry.class)
public void testGenY() {
    Assert.assertEquals("hello", "hallo");

}

}

The TestRetry.java

package cases;

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class TestRetry implements IRetryAnalyzer{

private int retryCount = 0;
private int maxRetryCount = 1;

@Override
public boolean retry(ITestResult arg0) {
    // TODO Auto-generated method stub
    if (retryCount < maxRetryCount) {
        retryCount++;
        return true;
    }

    return false;
}
}

Finally the XML. I right click it and run as the testNG suite.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite1" parallel="false" preserve-order="true">

    <test name="TestA">
        <classes>
            <class name="cases.RunTest" />
        </classes>
    </test> <!-- Test -->
        <listeners>
        <listener class-name="cases.CustomLinstener" />
    </listeners>
</suite> <!-- Suite -->

Answer 3

The documentation for TestNG's IRetryAnalyzer does not specify test reporting behavior:

Interface to implement to be able to have a chance to retry a failed test.

There are no mention of "retries" on http://testng.org/doc/documentation-main.html and searching across the entire testng.org site only returns links to the documentation of and references to IRetryAnalyzer (see site:testng.org retry - Google Search).

As there is no documentation for how a retried test is reported we cannot make many sound expectations. Should each attempt appear in the test results? If so, is each attempt except for the last attempt marked as a skip and the last as either a success or a failure? It isn't documented. The behavior is undefined and it could change with any TestNG release in subtle or abrupt ways.

As such, I recommend using a tool other than TestNG for retry logic.

e.g. You can Spring Retry (which can be used independently of other Spring projects):

TestRetry.java

public class TestRetry {
    private static RetryOperations retryOperations = createRetryOperations();

    private static RetryOperations createRetryOperations() {
        RetryTemplate retryTemplate = new RetryTemplate();
        retryTemplate.setRetryPolicy(createRetryPolicy());
        return retryTemplate;
    }

    private static RetryPolicy createRetryPolicy() {
        int maxAttempts = 2;
        Map<Class<? extends Throwable>, Boolean> retryableExceptions =
                Collections.singletonMap(AssertionError.class, true);
        return new SimpleRetryPolicy(maxAttempts, retryableExceptions);
    }

    @Test
    public void testGenX() {
        runWithRetries(context -> {
            Assert.assertEquals("google", "google");
        });
    }

    @Test
    public void testGenY() {
        runWithRetries(context -> {
            Assert.assertEquals("hello", "hallo");
        });
    }

    private void runWithRetries(RetryRunner<RuntimeException> runner) {
        retryOperations.execute(runner);
    }
}

RetryRunner.java

/**
 * Runner interface for an operation that can be retried using a
 * {@link RetryOperations}.
 * <p>
 * This is simply a convenience interface that extends
 * {@link RetryCallback} but assumes a {@code void} return type.
 */
interface RetryRunner<E extends Throwable> extends RetryCallback<Void, E> {
    @Override
    default Void doWithRetry(RetryContext context) throws E {
        runWithRetry(context);
        return null;
    }

    void runWithRetry(RetryContext context) throws E;
}

Console Output

===============================================
Default Suite
Total tests run: 2, Failures: 1, Skips: 0
===============================================

Spring Retry may look slightly more complicated at first but it provides very flexible features and API and enables separation of concerns of the test retry logic and the test reporting.