New data is not passed from TestNG DataProvider to test on test retry

Question

I am using TestNG @DataProvider for passing objects as test parameters and Retry class (that extends RetryAnalyzerCount) to retry failed tests.

All is working fine except: if some of object properties were changed during test before it fails, dataProvider method is running on retry but don't pass new object to test, so in test we have old set data from 1st run.

Example:

    @DataProvider(name = "userData")
    public static Object[] getUserData() {
        System.out.println("DataProvider is started...");
        TestUser testUser = new TestUser("John");
        System.out.println("Username from dataprovider: " + testUser.getName());
        return new Object[]{testUser};
    }

    @Test(retryAnalyzer = Retry.class, dataProvider = "userData")
    public void testRetry(TestUser user) {
        System.out.println("Test is started...");
        System.out.println("Username from test: " + user.getName());
        user.setName("New Name");
        Assert.fail();
    }

Output:

DataProvider is started...
Username from dataprovider: John
Test is started...
Username from test: John

Retry test...

DataProvider is started...
Username from dataprovider: John
Test is started...
Username from test: New Name

TestNG version: 7.6.0, 7.7.1

Facing such issue and don't know how to get around it.

Answer 1

The initial state should be maintained. This seems to have broken with 7.5 release. You can raise a bug or submit a PR. As a workaround you can shift to something lower if it is holding you.

Answer 2

There is no issue here.

• The test method is actually getting the same object as a parameter, when a retry happens.
• But what is causing the confusions is the fact that the data provider is being called even when a test method is being retried.
• TestNG is doing this, even though it never uses the object instantiated by the duplicate calls to the data provider. I have filed a issue that tracks this discrepancy of data provider getting called multiple times. Please track it here : https://github.com/cbeust/testng/issues/2884

Here's a sample that re-iterates all these aspects. For the sake of completeness I have added the same details to your post on the TestNG google users group as well.

Test class sample (I am using 7.7.1 which is the latest released version for running this test)

import com.rationaleemotions.DemoTest.Commentary;
import java.util.concurrent.atomic.AtomicInteger;
import org.testng.Assert;
import org.testng.IExecutionListener;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(Commentary.class)
public class DemoTest {

  public static final AtomicInteger dataProviderInvocationCount = new AtomicInteger(0);

  @Test(dataProvider = "dp", retryAnalyzer = TryAgain.class)
  public void testMethod(Pojo user) {
    System.out.println("Test is started for data " + user.toString() + " with name " + user.getName());
    user.setName("Dragon-Warrior");
    Assert.fail();
  }

  @DataProvider(name = "dp")
  public static Object[][] getTestData() {
    System.err.println(
        "Data Provider invocation count #" + dataProviderInvocationCount.incrementAndGet());
    return new Object[][]{
        {new Pojo().setName("John")}
    };
  }

  public static class TryAgain implements IRetryAnalyzer {

    private int counter = 1;

    @Override
    public boolean retry(ITestResult result) {
      return counter++ != 3;
    }
  }

  public static class Pojo {

    private String name;

    public Pojo() {
      System.err.println("Instantiated " + this);
    }

    public String getName() {
      return name;
    }

    public Pojo setName(String name) {
      this.name = name;
      return this;
    }
  }

  public static class Commentary implements IExecutionListener {

    @Override
    public void onExecutionStart() {}

    @Override
    public void onExecutionFinish() {
      System.err.println(
          "Data Provider was invoked " + dataProviderInvocationCount.get() + " time(s).");
    }
  }
}

Here's the complete execution output, along with some inline comments that I have added for elaboration:

Data Provider invocation count #1
Instantiated com.rationaleemotions.DemoTest$Pojo@2bfc268b
Test is started for data com.rationaleemotions.DemoTest$Pojo@2bfc268b with name John

Test ignored.
Data Provider invocation count #2
Instantiated com.rationaleemotions.DemoTest$Pojo@2b91004a
Test is started for data com.rationaleemotions.DemoTest$Pojo@2bfc268b with name Dragon-Warrior

Test ignored.
Data Provider invocation count #3
Instantiated com.rationaleemotions.DemoTest$Pojo@169e6180
Test is started for data com.rationaleemotions.DemoTest$Pojo@2bfc268b with name Dragon-Warrior

java.lang.AssertionError: null

    at org.testng.Assert.fail(Assert.java:110)
    at org.testng.Assert.fail(Assert.java:115)
    at com.rationaleemotions.DemoTest.testMethod(DemoTest.java:22)


===============================================
Default Suite
Total tests run: 3, Passes: 0, Failures: 1, Skips: 0, Retries: 2
===============================================

Data Provider was invoked 3 time(s).

Process finished with exit code 0

Points to note:

• The print statement in the data provider is getting invoked multiple times, re-iterating the point that the data provider is being called multiple times (whereas it should not have been the case).
• The instance Pojo@2bfc268b that was created the first time the data provider was called, is what is being passed to the test method when a retry is happening and so the name that was changed by me in the test method is what is coming back when the test is getting retried (which is why John became Dragon-Warrior)
• TestNG is ending up calling the data provider multiple times, which is causing the duplicate objects Pojo@2b91004a, Pojo@169e6180 to be created, but if you look at the execution output, you would notice that its not being used by the test method. This is a bug which is now being tracked by the above mentioned github issue.