When a @BeforeTest method fails, why is it not being listed in the testng-failed.xml?

Question

I am using maven with testng 6.14.3.

Here is my code structure:

testng.xml

<?xml version="1.0" encoding="UTF-8"?>
<suite name="set-3" parallel="tests" thread-count="10">
    <listeners>
        <listener class-name="org.uncommons.reportng.HTMLReporter" />
    </listeners>

    <test name="Customer Tests">
        <groups>
            <run>
                <include name="abc"/>
            </run>
        </groups>
        <classes>
            <class name="apps.Test1_BeforeTest_Of_Test2"></class>
            <class name="apps.Test2"></class>
        </classes>
    </test>

</suite>

Test1_BeforeTest_Of_Test2.java

public class Test1_BeforeTest_Of_Test2{
@BeforeTest(groups = {"abc"})
    public void test1Method() throws Exception {
}
@AfterTest(groups={"abc"})
public void test1AfterMethod() throws Exception {
    }
}

Test2.java

public class Test2{
 @Test(groups = {"abc"})
    public void test2Method(){
    }
}

During my run, Test1_BeforeTest_Of_Test2 class fails. So, Test2 is marked as skipped. But, when I look at the testng-failed.xml that is generated at the end of the run, the failed @BeforeTest class (Test1_BeforeTest_Of_Test2) is not included/listed:

testng-failed.xml

<?xml version="1.0" encoding="UTF-8"?>
<suite thread-count="10" name="Failed suite [set-3]" parallel="tests">
  <listeners>
    <listener class-name="org.uncommons.reportng.HTMLReporter"/>
  </listeners>
  <test name="Customer Tests(failed)">
    <groups>
      <run>
        <include name="abc"/>
      </run>
    </groups>
    <classes>
      <class name="apps.Test2">
        <methods>
          <include name="test2Method"/>
        </methods>
      </class> 
    </classes>
  </test> 
</suite>

Is this expected behaviour? Or a bug/gap in testng-failed.xml?

Ideally, when we re-run the failed tests, we expect the @BeforeTest to run as well, because it is pre-req for Test 2.

score 1 · Answer 1 · answered Jul 08 '21 at 05:06

TestNG currently seems to be honouring configurations to be considered in the testng-failed.xml if its part of the skipped test method's test class i.e., the configuration (which is perhaps what has caused a test to be skipped) needs to reside in the same java class as your skipped method for TestNG to consider it to be included.

In your example, that's not the case and the configuration method exists in a different test class (which is perfectly valid).

This looks like a bug in TestNG to me.

I have submitted a bug on your behalf on the TestNG project and will get it fixed in the upcoming version (7.5.0).

Defect : https://github.com/cbeust/testng/issues/2611

When a @BeforeTest method fails, why is it not being listed in the testng-failed.xml?

1 Answers1