Prevent TestNG from continuing suite if a test method does not exist

Question

This causes a lot of churn on debugging and I'm thinking there has to be a way to prevent this. Right now, if a test method does not exist (say it was misspelled), the suite will just skip that method and continue with the next one no issues. This causes lots of issues and it's difficult to find the reason. Here is an example:

    <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Sampoe_testSuite" preserve-order="true">
    <listeners>
        <listener class-name="framework.Listener"/>
    </listeners>
    <test name="Sample_TestSuite-Part1" preserve-order="true">
        <classes>
            <class name="tests.FirstTest">
                <methods>
                    <include name="firstMethod"/>
                </methods>
            </class>
            <class name="tests.SecondTest">
                <methods>
                    <include name="secondMethod"/>
                    <include name="thirdMethod"/>
                </methods>
            </class>
            <class name="tests.ThirdTest">
                <methods>
                    <include name="fourthMethod"/>
                </methods>
            </class>
        </classes>
    </test>
</suite>

Let's say that I misspelled the secondMethod of the SecondTest. It's actually sceondMethod in code. When I run this suite, there will be no errors, but what will happen is:

Runs FirstTest.firstMethod
Runs SecondTest.thirdMethod
Runs ThirdTest.fourthMethod

Notice that it just skips the misspelled method and continues on happily. I want it to fail the suite and say that there is a missing method. Is that possible?

Marcin · Answer 1 · 2020-11-10T19:24:10.043

You have to implement your own SuiteListener class implementing ISuiteListener. Implement onStart method which compares methods included in XML file with methods existing in test directory. If there are any, use System.exit(0) to stop executing.

import org.testng.*;
import org.testng.annotations.Test;
import org.testng.xml.XmlInclude;

import java.awt.*;
import java.io.File;
import java.io.IOException;
import java.lang.reflect.Method;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.Stream;

public class SuiteListener implements ISuiteListener {

    @Override
    public void onStart(ISuite suite) {
        // getting included method names in XML suite file
        List<String> inclduedMethodsInXmlSuite =
                suite.getXmlSuite().getTests().stream()
                        .flatMap(xmlTest -> xmlTest.getClasses().stream()
                                .flatMap(xmlClass -> xmlClass.getIncludedMethods().stream()
                                        .map(XmlInclude::getName)))
                        .collect(Collectors.toList());

        // getting all test methods
        List<String> testMethods = new ArrayList<>();
        try (Stream<Path> paths = Files.walk(Paths.get("./src/test/java"))) { // path to test classes directory
            testMethods = paths
                    .filter(path -> path.getFileName().toString().endsWith(".java")) // filtering only classes, not directories
                    .map(path -> path.getParent().getFileName().toString() + "." + path.getFileName().toString()) // mapping to qualified name, e.g test.TestClass
                    .map(file -> file.substring(0, file.length() - 5)) // removing ".java"
                    .map(clazz -> {
                                try {
                                    return Class.forName(clazz); // casting to Class object
                                } catch (ClassNotFoundException e) {
                                    e.printStackTrace();
                                    return null;
                                }
                            }
                    )
                    .flatMap(clazz -> Arrays.stream(clazz.getDeclaredMethods()) // getting methods of a test class annotated with @Test
                            .filter(method -> method.isAnnotationPresent(Test.class)))
                    .map(Method::getName) // mapping to method name
                    .collect(Collectors.toList());
        } catch (IOException e) {
            e.printStackTrace();
        }

        // checking if any of xml suite methods does not exist in test methods
        final List<String> finalTestMethods = testMethods;
        List<String> notSupprtedMethods = inclduedMethodsInXmlSuite.stream()
                .filter(xmlMethod -> !finalTestMethods.contains(xmlMethod))
                .collect(Collectors.toList());

        if (notSupprtedMethods.size() > 0) {
            System.out.println("Unsupported methods in XML Suite file:");
            notSupprtedMethods.forEach(System.out::println);
            System.exit(0);
        }

    }

    @Override
    public void onFinish(ISuite suite) {

    }
}

Couple of issues with this. One, java is a compiled language, thus there only exists a jar on the machine that executes the jar file, and all of the .java is complied to .class. Thus, there is no path that contains any .java on the machine that executes the test suites. Two, there could be dozens of java files with many dozen test methods in each. That could add a substantial amount of time to each run. — GregMa, Nov 11 '20 at 20:35

Prevent TestNG from continuing suite if a test method does not exist

1 Answers1