How can I tell Gradle to use my testng.xml file for Test Classes and Ordering?

Question

I am attempting to have Gradle execute some tests defined using a testng.xml file. My testng.xml file looks like this:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="mySuite" verbose="1">

  <listeners>
    <listener class-name="mypackage.MyListener" />
    <listener class-name="mypackage.TestOrderer" />
  </listeners>

  <test name="Tests">
    <classes>
      <class name="mytestpackage.CrazyTest1"/>
      <class name="mytestpackage.CrazyTest2"/>
      <class name="mytestpackage.CrazyTest3"/>
    </classes>
  </test>
</suite>

So why do I need this? I want to ensure that my tests are organized in a way that's defined by annotations similar to that which was listed here. As you might guess, TestOrderer is an IMethodInterceptor.

Here's the problem, Gradle seems to be taking over my testng.xml file and scraping the test directory to find the tests it wants to execute. Even if I disable this, it fails to execute the methods appropriately. Here's what I think should work, but just, doesn't.

test {
  useTestNG()
  options.suites("src/test/resources/crazyTestNG.xml") 
  # Adding 
  # scanForTestClasses = false 
  # causes zero tests to be executed, since the class names don't end in Test
}

It seems like it should be a no-brainer...fork the TestNG process and let it run, but I can't figure out how to tell Gradle to get out of the way and just execute my tests.

try changing the syntax to `options { closure }` per docs at https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:options — MarkHu, Aug 04 '22 at 22:21

score 15 · Answer 1 · answered Jun 16 '15 at 16:17

15

Here's how you can configure a test suite (xml) to be executed in a Gradle task:

apply plugin: 'java'

repositories {
    mavenCentral()
}
dependencies {
    // add the dependencies as needed
    testCompile group: 'org.testng', name: 'testng', version:'6.8.8'
    testCompile fileTree('lib')
}
test {
    useTestNG() {
        // runlist to executed. path is relative to current folder
        suites 'src/test/resources/runlist/my_test.xml'
    }
}

answered Jun 16 '15 at 16:17

Henrique Gontijo

1,052
2
15
27

More options can be found [here](https://docs.gradle.org/current/groovydoc/org/gradle/api/tasks/testing/testng/TestNGOptions.html). – Nick Aug 19 '15 at 03:25
2

@Nick That link is dead now but you were thinking of something on this page perhaps? https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html – Per Lundberg Oct 31 '18 at 13:28
1

Aug 19 2015 was Gradle 2.6 so he was probably referring to https://docs.gradle.org/2.6/groovydoc/org/gradle/api/tasks/testing/testng/TestNGOptions.html and the current (5.6.2 at time of writing) is now here https://docs.gradle.org/current/javadoc/org/gradle/api/tasks/testing/testng/TestNGOptions.html – opticyclic Sep 17 '19 at 21:51
1

to parameterize, replace the `suites 'hardcoded/path'` with `suites project.getProperty("suite")` --more details at https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#N317DF – MarkHu Aug 04 '22 at 22:11

score 11 · Answer 2 · edited Mar 05 '15 at 01:52

I hated the TestNG support in gradle... Found it to be most un-flexable compared to using raw TestNG. And I was tired of fiddling with gradle. My solution.. Run TestNG directly using a Gradle task

task runTests(type: JavaExec, dependsOn: 'classes') {
    main = 'org.testng.TestNG'
    classpath = files("./src/test/resources",
                      project.sourceSets.main.compileClasspath,
                      project.sourceSets.test.compileClasspath,
                      project.sourceSets.main.runtimeClasspath,
                      project.sourceSets.test.runtimeClasspath)
    args = ["-parallel",  "methods", "-threadcount", "1", "-d", "./build/test-output", "./src/test/resources/test.xml"]
}

Which I run from the command line:

gradle runTests

score 2 · Answer 3 · answered Sep 30 '20 at 12:09

As another answer points out: the solution is to use the suites command. Though I prefer parameterizing the argument for that command so that from the command line I can choose any TestNG suite I want to run.

test {
    
    // Detect if suite param was passed in
    def runSuite = project.hasProperty("suite")
    
    useTestNG() {
        if (runSuite) {
            // If parameter was passed in, use it in the 'suites' command
            def suiteToRun = project.getProperty("suite")
            suites "src/test/resources/"+suiteToRun
        } else {
            // Handle non-command line executions e.g. running tests from IDE
            parallel 'methods'
            threadCount 2
        }
    }
}

Then from the command line I can run something like:

gradle test -Psuite=mysuite.xml

I prefer this to defining a bunch of custom Gradle tasks since that approach results in a messy build.gradle file and is slightly less flexible to new suites being added.

djangofan · Answer 4 · 2013-02-09T21:17:52.250

This method doesn't use your testng.xml file but you could also emulate testNG test groups and order by creating JUnit test groups as Gradle tasks and then order them manually by ordering the task execution when you execute your build:

// in build.gradle
task testGroupOne(type: Test) {
   //include '**/*SuiteOne.*'
   include '**/SuiteOne.class'
   testReportDir = file("${reportsDir}/SuiteOne")
   testResultsDir = file("${buildDir}/test-results/SuiteOne")
}

task testGroupTwo(type: Test) {
   //include '**/*SuiteTwo.*'
   include '**/SuiteTwo.class'
   testReportDir = file("${reportsDir}/SuiteTwo")
   testResultsDir = file("${buildDir}/test-results/SuiteTwo")
}

Then, run your build like: gradle testGroupTwo testGroupOne

score 1 · Answer 5 · answered Jan 22 '11 at 04:39

The Gradle TestNG runner assumes that if no test classes are specified, either by scanning for them, or pattern matching on the name, then it should skip test execution entirely.

Instead, it should consider whether a suite xml has been provided or not as well. Could you add a jira issue for this problem?

One possible work around is to use options.listener to specify the listeners, and not use a suite xml file at all:

test {
   options.listeners << 'mypackage.MyListener'
   options.listeners << 'mypackage.TestOrderer'
}

This way you don't need to specify the test classes, and you can just let the scanning find them for you.

No I need to specify the test classes. I need to get them to run in a very specific order due to a specific use case of the TestNG engine. Scanning breaks this process. — Mike Cornell, Jan 24 '11 at 13:20

score 0 · Answer 6 · answered Apr 28 '22 at 07:26

task runTests(type: JavaExec, dependsOn: 'classes') {
    main = 'org.testng.TestNG'
    classpath = files("./src/test/resources",
                      project.sourceSets.main.compileClasspath,
                      project.sourceSets.test.compileClasspath,
                      project.sourceSets.main.runtimeClasspath,
                      project.sourceSets.test.runtimeClasspath)
    args = ["-parallel",  "methods", "-threadcount", "1", "-d", "./build/test-output", "./src/test/resoenter code hereurces/test.xml"]
}

How to pass VM arguments in args [] ?

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). — Community, May 03 '22 at 04:25

How can I tell Gradle to use my testng.xml file for Test Classes and Ordering?

6 Answers6

Linked