Jacoco test coverage in Gradle custom tasks

Question

At my current project we Spring Boot + Kotlin app with a huge test suite, but a fraction of those tests are flaky. So, we'd like to have those flaky tests run separately from the healthy tests, to try and give devs faster feedback about healthy tests that break.

At the same time, we have the goal of pursuing 100% test coverage and we do that with Jacoco & Gradle.

The problem we're facing is that when we run ./gradlew test flakyTests jacocoTestCoverageVerification, the test results from test & flakyTests don't seem to be "accumulated", meaning that Jacoco only sees a subset of the test results and reports a violation, saying that the test coverage is below the 1.0 threshold.

Now here comes the question: is there a way to make Jacoco understand that the results should be combined when calculating the test coverage indices?

This is roughly how tests and jacoco are setup on build.gradle:

test {
    useJUnitPlatform()
    filter {
        excludeTestsMatching('*Flaky*')
    }
    testLogging {
        events "passed", "skipped", "failed"
        exceptionFormat = 'full'
    }
}

task flakyTests(type: Test) {
    description = 'Runs flaky / unstable tests.'
    useJUnitPlatform()
    filter {
        includeTestsMatching('*Flaky*')
    }
    testLogging {
        events "passed", "skipped", "failed"
        exceptionFormat = 'full'
    }
}


jacoco {
    toolVersion = "0.8.7"
}

jacocoTestReport {
    afterEvaluate {
        classDirectories.setFrom(files(classDirectories.files.collect {
            fileTree(dir: it, exclude: testReportExclusions)
        }))
    }
    reports {
        xml.setEnabled(true)
        html.setEnabled(true)
        html.destination file("${buildDir}/jacocoHtml")
    }
}

jacocoTestCoverageVerification {
    afterEvaluate {
        classDirectories.setFrom(files(classDirectories.files.collect {
            fileTree(dir: it, exclude: testReportExclusions)
        }))
    }
    violationRules {
        rule {
            limit {
                counter = 'INSTRUCTION'
                minimum = 1.0
            }
            limit {
                counter = 'BRANCH'
                minimum = 1.0
            }
            limit {
                counter = 'LINE'
                minimum = 1.0
            }
            limit {
                counter = 'METHOD'
                minimum = 1.0
            }
            limit {
                counter = 'CLASS'
                minimum = 1.0
            }
        }
    }
}

Thanks a lot for any help!

score 0 · Answer 1 · answered Jun 29 '21 at 18:50

All we had to do was adding a executionData instruction, this is how it ended up:

jacocoTestCoverageVerification {
    executionData tasks.withType(Test).findAll { it.state.executed }
    afterEvaluate {
        classDirectories.setFrom(files(classDirectories.files.collect {
            fileTree(dir: it, exclude: testReportExclusions)
        }))
    }
    violationRules {
        rule {
            limit {
                counter = 'INSTRUCTION'
                minimum = 1.0
            }
            limit {
                counter = 'BRANCH'
                minimum = 1.0
            }
            limit {
                counter = 'LINE'
                minimum = 1.0
            }
            limit {
                counter = 'METHOD'
                minimum = 1.0
            }
            limit {
                counter = 'CLASS'
                minimum = 1.0
            }
        }
    }
}

Jacoco test coverage in Gradle custom tasks

1 Answers1