Gradle: guidance needed to filter tests using annotation

Question

We are developing a large test suite which is intended to run on multiple environments, such as smoke, performance, full suite, etc. (We are currently using JUnit as the test framework.) What we are trying to achieve is annotate tests (classes or methods or both) with one or more annotations like @SmokeTest, @PerformanceTest, @AcceptanceTest and then add some test tasks in the build.gradle to run a specific selection of tests based on those annotations. The idea is very much like this one from the Gradle forum.

I know Gradle can detect JUnit tests based on JUnit annotations (see 23.12.4. Test detection in the gradle user guide). However, I cannot find out how I can leverage that capability and add some custom logics of my own. What I am looking for is something like below:

1. Detect the tests based on one or more given annotations (include or exclude)
2. Add the detected tests into a container of some sort.
3. Have a test task to run tests in the container.

So, I would like to ask any guidance that you can provide to achieve that functionality. Thank you very much.

Answer 1

After some research, I think I have solution now. I created a demo project at github. (The name of the project is a little misleading :-]).

I also would like to paste the core logic of the solution here, which is extracted from the build.gradle in the demo project:

List testClassNames() {
    File classesDir = sourceSets.test.output.classesDir
    String prefix = classesDir.toString() + '/'
    List names = []
    classesDir.traverse {
        if( it.absolutePath.endsWith( '.class' ) ) {
            String name = (it.absolutePath - prefix).replaceAll( '/', '.' ) - '.class'
            names << name
        }
    }
    return names
}

ClassLoader getClassLoader() {
    List urls = sourceSets.test.runtimeClasspath.collect {
        it.toURI().toURL()
    }
    return URLClassLoader.newInstance( urls as URL[] )
}

List annotationFilter( Map map ) {
    map.prefix = map?.prefix ?: '' // prefix: provide convenience for passing in annotation names

    ClassLoader loader = classLoader

    List result

    // filter with annotations
    if( !map.includes ) {
        result = map?.names
    } else {
        result = []
        map?.names.each { name ->
            Class klass = loader.loadClass( name )
            map?.includes.each { annotationName ->
                String fullName = map.prefix + annotationName
                Class<? extends Annotation> annotation = loader.loadClass( fullName ).asSubclass( Annotation )
                if( klass.isAnnotationPresent( annotation ) ) {
                    result << name
                }
            }
        }
    }

    if( result?.size() == 0 ) result = [ 'no.tests.to.run' ]
    return result
}

task smoke( type: Test, description: 'Run smoke tests' ) {
    doFirst {
        List names = testClassNames()
        List filtered = annotationFilter( names: names, includes: ['demo.Smoke'] )
        println 'Running ' + filtered.size() + ' tests:\n' + filtered*.toString()*.replaceAll('^','\t').join('\n')

        filter {
            setIncludePatterns( filtered as String[] )
        }
    }
}

As you can see above, the smoke task will only execute tests annotated with @Smoke.

Answer 2

As mention in this answer, using Spoon and the corresponding Gradle plugin might be a more generic solution.

Edit: There also seems to be another Gradle plugin for Spoon here.