Custom test method name in TestNG reports

Question

I am working on a project where I need to invoke TestNG programatically(using data providers). Things are fine except that in the report, we are getting the name of the @Test method, which is a generic one to handle many cases. What we would like is to get a meaningful name in the report.

I was researching on this and found 3 ways, but unfortunately, all are failing for me.

1) Implement ITest

I have found about this here and here

I am setting the name I want as soon as I enter the @Test method(For all 3 ways i tried,this is how I am setting the name).This name is returned from getTestName(). What i observed is that getTestName() is getting called before and after my @Test. Initially, it is returning null(for handling NullPointerException, I return "" instead of null) and later it returns correct value. But i dont see this getting reflected in the report

Edit:Also tried setting the name from@BeforeMethod as suggested by artdanil

2 and 3

Both are based on solutions given in the second link above

By overriding setName in XmlSuite, I am getting

Exception in thread "main" java.lang.AssertionError: l should not be null
        at org.testng.ClassMethodMap.removeAndCheckIfLast(ClassMethodMap.java:58)
        at org.testng.internal.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:208)
        at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:114)
        at org.testng.TestRunner.privateRun(TestRunner.java:767)
        ...

By overriding toString(), I see these in logs (with my comments) but no updates in report

[2013-03-05 14:53:22,174] (Main.java:30) - calling execute 
    [2013-03-05 14:53:22,346] GenericFunctionTest.<init>(GenericFunctionTest.java:52) - inside constructor
    [2013-03-05 14:53:22,372] GenericFunctionTest.toString(GenericFunctionTest.java:276) - returning **//this followed by 3 invocations before arriving at @Test method**
    [2013-03-05 14:53:22,410] GenericFunctionTest.toString(GenericFunctionTest.java:276) - returning 
    [2013-03-05 14:53:22,416] GenericFunctionTest.toString(GenericFunctionTest.java:276) - returning 
    [2013-03-05 14:53:22,455] GenericFunctionTest.toString(GenericFunctionTest.java:276) - returning 
    [2013-03-05 14:53:22,892] GenericFunctionTest.<init>(GenericFunctionTest.java:52) - inside constructor 
    [2013-03-05 14:53:23,178] GenericFunctionTest.toString(GenericFunctionTest.java:276) - returning **//again blank as i havent set it yet**
    [2013-03-05 14:53:23,182] GenericFunctionTest.getResult(GenericFunctionTest.java:69) - inside with test case:TestCase{signature=Signature{...}}**//I am setting it immedietely after this**
    [2013-03-05 14:53:23,293] GenericFunctionTest.toString(GenericFunctionTest.java:276) - returning MyMethodName **//What i want**
    [2013-03-05 14:53:23,299] GenericFunctionTest.toString(GenericFunctionTest.java:276) - returning MyMethodName **// again**

Edit: tried again all 3 by hardcoding a value rather than setting it on entry of my test method. But same results

Answer 1

I had the same problem, and found that it helps to set the field storing test case name in the method annotated with @BeforeMethod, using native injection of TestNG to provide method name and test parameters. The test name is taken from test parameters supplied by the DataProvider. If your test method does not have parameters, just report the method name.

//oversimplified for demontration purposes
public class TestParameters {
    private String testName = null;
    private String testDescription = null;

    public TestParameters(String name,
                          String description) {
        this.testName = name;
        this.testDescription = description;
    }

    public String getTestName() {
        return testName;
    }
    public String getTestDescription() {
        return testDescription;
    }
}

public class SampleTest implements ITest {
    // Has to be set to prevent NullPointerException from reporters
    protected String mTestCaseName = "";

    @DataProvider(name="BasicDataProvider")
    public Object[][] getTestData() {
        Object[][] data = new Object[][] {
                { new TestParameters("TestCase1", "Sample test 1")},
                { new TestParameters("TestCase2", "Sample test 2")},
                { new TestParameters("TestCase3", "Sample test 3")},
                { new TestParameters("TestCase4", "Sample test 4")},
                { new TestParameters("TestCase5", "Sample test 5") }
        };
        return data;
    }

    @BeforeMethod(alwaysRun = true)
    public void testData(Method method, Object[] testData) {
        String testCase = "";
        if (testData != null && testData.length > 0) {
            TestParameters testParams = null;
            //Check if test method has actually received required parameters
            for (Object testParameter : testData) {
                if (testParameter instanceof TestParameters) {
                    testParams = (TestParameters)testParameter;
                    break;
                }
            }
            if (testParams != null) {
                testCase = testParams.getTestName();
            }
        }
        this.mTestCaseName = String.format("%s(%s)", method.getName(), testCase);
    }

    @Override
    public String getTestName() {
        return this.mTestCaseName;
    }

    @Test(dataProvider="BasicDataProvider")
    public void testSample1(TestParameters testParams){
        //test code here
    }

    @Test(dataProvider="BasicDataProvider")
    public void testSample2(TestParameters testParams){
        //test code here
    }

    @Test
    public void testSample3(){
        //test code here
    }
}

EDIT: Based on the comments below, I realized a sample from report will be useful.

Extract from the report from running code above:

<testng-results skipped="0" failed="0" total="5" passed="5">
  <suite name="SampleTests" duration-ms="2818" started-at="<some-time>" finished-at="<some-time>">
    <test name="Test1" duration-ms="2818" started-at="<some-time>" finished-at="<some-time>">
        <test-method 
            status="PASS" 
            signature="testSample1(org.example.test.TestParameters)[pri:0, instance:org.example.test.TimeTest@c9d92c]"
            test-instance-name="testSample1(TestCase5)"
            name="testSample1" 
            duration-ms="1014"
            started-at="<some-time-before>" 
            data-provider="BasicDataProvider" 
            finished-at="<some-time-later>" >
            <!-- excluded for demonstration purposes -->
        </test-method>
        <!-- the rest of test results excluded for brevity -->
    </test>
  </suite>
</testng-result>

Note, that the value returned from getTestName() method is in the test-instance-name attribute, and not in the name attribute.

Answer 2

I ran into a similar problem. First I implemented the ITest strategy already mentioned. And this is part of the solution, but not completely.

TestNG, for some reason, when building different reports, calls getName() on the test while building the report. This is fine if you are not using a data provider to generate different runs and set a unique name for each run by using the ITest strategy. If you are using a data provider to generate multiple runs of the same test and want each run to have a unique name then there is a problem. As the ITest strategy leaves the name for the test as the name set by the last run.

So I had to implement a very custom getName(). SOme assumptions (in my particular case):

1. Only three reports are run: TestHTMLReporter, EmailableReporter, XMLSuiteResultWriter.
2. When ever get name is not called as a result of one of the assumed reporters, then returning the currently set name is fine.
3. When a reporter is running, it makes its getName() calls in order and only 1 time for each run.

public String getTestName()
{
    String name = testName;
    StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace();//.toString();
    if(calledFrom(stackTrace, "XMLSuiteResultWriter"))
    {
        name = testNames.size()>0?testNames.get(xmlNameIndex<testNames.size()?xmlNameIndex:0):"undefined";
        xmlNameIndex++;
        if(xmlNameIndex>=testNames.size())
            xmlNameIndex = 0;
    }
    else if(calledFrom(stackTrace, "EmailableReporter"))
    {
        name = testNames.size()>0?testNames.get(emailNameIndex<testNames.size()?emailNameIndex:0):"undefined";
        emailNameIndex++;
        if(emailNameIndex>=testNames.size())
            emailNameIndex = 0;
    }
    if(calledFrom(stackTrace, "TestHTMLReporter"))
    {
        if(testNames.size()<0)
        {
            name = "undefined";
        }
        else
        {
            if(htmlNameIndex < testNamesFailed.size())
            {
                name = testNamesFailed.get(htmlNameIndex);
            }
            else
            {
                int htmlPassedIndex = htmlNameIndex - testNamesFailed.size();
                if(htmlPassedIndex < testNamesPassed.size())
                {
                    name = testNamesPassed.get(htmlPassedIndex);
                }
                else
                {
                    name = "undefined";
                }
            }
        }
        htmlNameIndex++;
        if(htmlNameIndex>=testNames.size())
            htmlNameIndex = 0;
    }
    return name;
}

private boolean calledFrom(StackTraceElement[] stackTrace, String checkForMethod)
{
    boolean calledFrom = false;
    for(StackTraceElement element : stackTrace)
    {
        String stack = element.toString();
        if(stack.contains(checkForMethod))
            calledFrom = true;
    }
    return calledFrom;
}

When setting the name for the run (I did this in the @BeforeMethod(alwaysRun=true) method I defined following the ITest strategy) I added the name to an ArrayList testNames. But then the html report was not correct. Most of the other reports pulls the information in order, like the XMLSuiteResultWriter, but TestHTMLReporter gets the name by first getting all the names for failed tests and then the names for passing tests. So I had to implement to additional ArrayLists: testNamesFailed and testNamesPassed and add the test names to them when the test finished based on whether they passed or not.

And I will freely admit this is very much a hack and very fragile. Ideally, TestNG adds the tests to a collection while running, and gets the name from that collection instead of from the original test. If you have TestNG to run other reports you will have to figure out what order they request the names and what is a unique enough string to search for in the thread stack trace.

--Edit 1

Alternatively, use the ITest Strategy and the factory pattern (@factory annotations).

TestNG Using @Factory and @DataProvider

http://beust.com/weblog/2004/09/27/testngs-factory/

It does require some minor changes. This includes creating a constructor with the same parameters as the original test method. The test method now has no parameters. You can set the name in the new constructor and simply return that in the getTestName method. Make sure to remove the data provider specification from the test method.

Answer 3

If you want to change the name in the HTML report, it'll be a total hack. Here's how I did it:

public class NinjaTest {
...
...
@AfterMethod (alwaysRun = true)
public void afterMethod(ITestResult result, Method method) {
    try {
        //I have XML test suites organized in directories. 
        String xmlFile = result.getTestContext().getCurrentXmlTest().getSuite().getFileName();
        String suiteName = xmlFile.substring(xmlFile.lastIndexOf("\\") + 1, xmlFile.lastIndexOf(".xml"));
        String pathToFile = xmlFile.substring(0, xmlFile.lastIndexOf("\\") );
        String directory = pathToFile.substring(pathToFile.lastIndexOf("\\") + 1);
        String testMethodName = String.format("%s/%s - %s", directory, suiteName, method.getName());

        //Total hack to change display name in HTML report  ＼(^o^)／ 
        Field methodName = org.testng.internal.BaseTestMethod.class.getDeclaredField("m_methodName");
        methodName.setAccessible(true);
        methodName.set(result.getMethod(), testMethodName);
    } catch (Exception e) {
        // Eh....  ¯\_(ツ)_/¯
        e.printStackTrace();
    }
}
...
...

Answer 4

Please find following code for set custom name of testcase in TestNG reports.

Following features are available in this code.

• Dynamic execution on same test-case in multiple time
• Set custom test-case name for reports

• Set parallel execution of multiple test-cases execution

  import java.lang.reflect.Field;
  import org.testng.ITest;
  import org.testng.ITestResult;
  import org.testng.Reporter;
  import org.testng.annotations.AfterMethod;
  import org.testng.annotations.DataProvider;
  import org.testng.annotations.Factory;
  import org.testng.annotations.Test;
  import org.testng.internal.BaseTestMethod;
  import com.test.data.ServiceProcessData;
  
  public class ServiceTest implements ITest {
  
  protected ServiceProcessData serviceProcessData;
  protected String testCaseName = "";
  
  @Test
  public void executeServiceTest() {
      System.out.println(this.serviceProcessData.toString());
  }
  
  @Factory(dataProvider = "processDataList")
  public RiskServiceTest(ServiceProcessData serviceProcessData) {
      this.serviceProcessData = serviceProcessData;
  }
  
  @DataProvider(name = "processDataList", parallel = true)
  public static Object[] getProcessDataList() {
  
      Object[] serviceProcessDataList = new Object[0];
      //Set data in serviceProcessDataList
      return serviceProcessDataList;
  }
  
  @Override
  public String getTestName() {
  
      this.testCaseName = "User custom testcase name";
  
      return this.testCaseName;
  }
  
  @AfterMethod(alwaysRun = true)
  public void setResultTestName(ITestResult result) {
      try {
          BaseTestMethod baseTestMethod = (BaseTestMethod) result.getMethod();
          Field f = baseTestMethod.getClass().getSuperclass().getDeclaredField("m_methodName");
          f.setAccessible(true);
          f.set(baseTestMethod, this.testCaseName);
      } catch (Exception e) {
          ErrorMessageHelper.getInstance().setErrorMessage(e);
          Reporter.log("Exception : " + e.getMessage());
      }
  }}
  

  Thanks

Answer 5

The answer by artdanil did not completely solve my problem, the test name is not updated in emailable report.

Answer posted by @jersey-city-ninja does update the name in Emailable report but it repeats the same-last updated testname for all the Dataprovider values because what pilotg2 posted is true for tests that consume Dataprovider i.e. the getTestName method keeps returning the last set name for a method and all the testnames for a dataprovider are same.

So here is the answer that is combination of answer posted by @pilotg2 and @jersey-city-ninja and with additional step to overcome duplicate method names.

Note this updates the testname in Emailable report, XML report, HTML report, Junit report. I don't see it updating the Eclipse - TestNg execution view - will update if I found something

import org.testng.Assert;
import org.testng.ITest;
import org.testng.ITestContext;
import org.testng.ITestResult;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;


public class NewDataProviderTest implements ITest {
    //The Java ThreadLocal class enables you to create variables that can only be read and written by the same thread
    private ThreadLocal<String> testName = new ThreadLocal<>();

    /*
    * TestNG, for some reason, when building different reports, calls getName() on the test while building the report.
    * This is fine if you are not using a data provider to generate different runs and set a unique name for each run by using the ITest strategy.
    * If you are using a data provider to generate multiple runs of the same test and want each run to have a unique name then there is a problem.
    * As the ITest strategy returns the name for the test as the name set by the last run.
    * */
    private int emailNameIndex = 0;
    private int htmlNameIndex = 0;
    private int xmlNameIndex = 0;
    private ArrayList<String> allTests = new ArrayList<String>();
    /*
    * TestHTMLReporter gets the name by first getting all the names for failed tests and then the names for passing tests
    * Hence keeping them in 2 separate lists
    * */
    private ArrayList<String> passedTests = new ArrayList<String>();
    private ArrayList<String> failedTests = new ArrayList<String>();

    @BeforeClass(alwaysRun = true)
    public void initialize() {
        this.testName.set("");
    }

    @BeforeMethod(alwaysRun = true)
    public void setCustomTestcaseName(Method method, Object[] testData) {
        //Set the default name
        this.testName.set(method.getName());
        //Change the test name only if Dataprovider is used
        //Check if data provider is used in the test
        if (testData != null && testData.length > 0) {
            System.out.println("\n\nParameters "+testData[0]+" are passed to the test - "+method.getName());
            //Taking 1st field in the Dataprovider for updating testname - can be changed as desired maybe using a variable
            //I'm changing the name only if the Dataprovider field is String
            if (testData[0] instanceof String) {
                //Taking 1st field in the Dataprovider for updating testname - can be changed as desired
                System.out.println("I'm setting custom name to the test as "+method.getName() + "_" + testData[0]);
                this.testName.set(method.getName() + "_" + testData[0]);
            }

        }
        //Add the name to the collection that stores all list names
        allTests.add(testName.get());

    }

     @AfterMethod (alwaysRun = true)
     public void setTheTestcaseNameInResult(ITestResult result, Method method) {
        //Fill the Passed and Failed tests collections
         try {
             if(result.getStatus() == ITestResult.SUCCESS) {
                 System.out.println("Adding "+ result.getTestName() + " to passed tests collection");
                 passedTests.add(result.getTestName());
             }
             if(result.getStatus() == ITestResult.FAILURE) {
                 System.out.println("Adding " + result.getTestName() + " to FAILURE tests collection");
                 failedTests.add(result.getTestName());
             }
         } catch (Exception e) {
             e.printStackTrace();
         }
         // To change display name in HTML report
         //Only changing the name if the parameter is instance of String
         if(iTestResult.getParameters().length > 0) {
             if (iTestResult.getParameters()[0] instanceof String) {
                 System.out.println("Updating the name as Parameters are passed to the test-"+method.getName());
                 try {
                     /* This helps in setting unique name to method for each test instance for a data provider*/
                     Field resultMethod = TestResult.class.getDeclaredField("m_method");
                     resultMethod.setAccessible(true);
                     resultMethod.set(iTestResult, iTestResult.getMethod().clone());

                     Field methodName = org.testng.internal.BaseTestMethod.class.getDeclaredField("m_methodName");
                     methodName.setAccessible(true);
                     methodName.set(iTestResult.getMethod(), this.getTestName());
                 } catch (Exception e) {
                     e.printStackTrace();
                 }
                 System.out.println("New Name is - " + iTestResult.getMethod().getMethodName());

             }
         }
     }

    @Override
    public String getTestName() {
        String name = testName.get();
        StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace();// .toString();
        //This is called
        if (isCalledFromMethod(stackTrace, "XMLSuiteResultWriter")) {
            //System.out.println("Got called from XMLSuiteResultWriter");
            if (allTestNames.size() > 0) {
                if (xmlNameIndex < allTestNames.size()) {
                    name = allTestNames.get(xmlNameIndex);
                } else {
                    name = allTestNames.get(0);
                }
            } else {
                name = "undefined";
            }
            xmlNameIndex++;
            if (xmlNameIndex >= allTestNames.size()) {
                xmlNameIndex = 0;
            }
            // System.out.println("Got called from XMLSuiteResultWriter returning name - "+name);
        } else if (isCalledFromMethod(stackTrace, "EmailableReporter")) {
            if (allTestNames.size() > 0) {
                if (emailNameIndex < allTestNames.size()) {
                    name = allTestNames.get(emailNameIndex);
                } else {
                    name = allTestNames.get(0);
                }
            } else {
                name = "undefined";
            }
            emailNameIndex++;
            if (emailNameIndex >= allTestNames.size()) {
                emailNameIndex = 0;
            }
            System.out.println("Got called from EmailableReporter returning name -"+name);
        }
        if (isCalledFromMethod(stackTrace, "TestHTMLReporter")) {
            if (allTestNames.size() <= 0) {
                name = "undefined";
            } else {
                if (htmlNameIndex < failedTestNames.size()) {
                    name = failedTestNames.get(htmlNameIndex);
                } else {
                    int htmlPassedIndex = htmlNameIndex - failedTestNames.size();
                    if (htmlPassedIndex < passedTestNames.size()) {
                        name = passedTestNames.get(htmlPassedIndex);
                    } else {
                        name = "undefined";
                    }
                }
            }
            htmlNameIndex++;
            if (htmlNameIndex >= allTestNames.size()) {
                htmlNameIndex = 0;
            }
            System.out.println("Got called from TestHTMLReporter returning name - "+name);
        }
        System.out.println("Returning testname as-"+name);
        return name;
    }

    private boolean isCalledFromMethod(StackTraceElement[] stackTrace, String checkForMethod) {
        boolean calledFrom = false;
        for (StackTraceElement element : stackTrace) {
            String stack = element.toString();
            // System.out.println("Rohit the called from value is:"+stack);
            if (stack.contains(checkForMethod))
                calledFrom = true;
        }
        return calledFrom;
    }


  @Test(groups= {"L1", "L2", "L3"}, dataProvider = "dp1")
  public void dataProviderTest(String username) {
      System.out.println("\n\nI'm in dataProviderTest with data-"+username);
      /* Fail the test if value is L2 - deliberately so that we have failed test in report */ 
      if(username.contains("L2")) {
          Assert.fail();
      }

  }

  @Test(dependsOnMethods = "dataProviderTest", groups= {"L1", "L2", "L3"}, dataProvider = "dp1")
  public void dataProviderDependentTest(String username) {
      System.out.println("\n\nI'm in dataProvider DEPENDENT Test with data-"+username);

  }

  //This test consumes data of type list so the name will not be updated in report
  @Test(groups= {"L1", "L2", "L3"}, dataProvider = "dp2")
  public void dataListProviderTest(List<String[]> list) {
      Object[] arr = list.get(0);
        List<Object> arrList = Arrays.asList(arr);
        Iterator<Object> iterator = arrList.iterator();
        while (iterator.hasNext()) {

        String[] data = (String[]) iterator.next();
        System.out.println("In list test - "+data[0]);
        }    

  }

  @DataProvider(name="dp1")
  public Object[][] getDataForTest(ITestContext iTestContext){
      Object[][] L1 = new Object[][] {
          {"L1"}, {"L2"}, {"L3"}
      };


      return L1;
  }


  @DataProvider(name="dp2")
  public Object[][] getDataListForTest(ITestContext iTestContext){
      List<Object[][]> list = new ArrayList<Object[][]>();
      Object[][] L1 = new Object[][] {
          new String [] {"L1", "l1"}, 
          new String [] {"L1", "l1"}
      };

     list.add(L1);
      return new Object[][] { { list } };
  }

}

Answer 6

Try implementing the org.testng.ITest interface that requires a getTestName() method. This way reporting handles the returned value properly.

Answer 7

Running into the same issue this is how we solved it:

The problem is that all ITestResult result objects share a single ITestNGMethod object instance, so when the method name is changed it affects all results.

From: org.testng.reporters.XMLSuiteResultWriter#getTestResultAttributes

    attributes.setProperty(XMLReporterConfig.ATTR_NAME, testResult.getMethod().getMethodName());

The name attribute is being read from testResult.getMethod().getMethodName()

Basically we cloned the method object for each result, so each can have an independent getMethod().

We added this code:

  @AfterMethod(alwaysRun = true)
public void setResultTestName(ITestResult result, ITestContext context) {
    try {
        BaseTestMethod baseTestMethod = (BaseTestMethod) result.getMethod().clone();

        String featureName = StringUtils.substringBetween(result.getParameters()[1].toString(), "[", "]");
        String scenarioName = result.getParameters()[0].toString().replace("\"", "");
        Field f = baseTestMethod.getClass().getSuperclass().getDeclaredField("m_methodName");
        f.setAccessible(true);
        f.set(baseTestMethod, featureName + " - " + scenarioName);

        f = result.getClass().getDeclaredField("m_method");
        f.setAccessible(true);
        f.set(result, baseTestMethod);

    } catch (Exception e) {
        e.printStackTrace();
    }
}