How to make Jenkins stage fail and turn red when coverage is too low (using C# and dotnet test)?

Question

I am using C#, coverlet.msbuild and the Jenkins Cobertura adapter. I have roughly this in my Jenkinsfile:

stage ('Run unit tests') {
    steps {
        powershell "dotnet test -c:Release /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura --no-build --no-restore --logger trx"
    }
    post {
        always {
            step([$class: 'MSTestPublisher'])
            publishCoverage failUnhealthy: true, 
                globalThresholds: [[thresholdTarget: 'Package', unhealthyThreshold: 50.0]],
                adapters: [coberturaAdapter(
                    mergeToOneReport: true, 
                    path: '**/*.cobertura.xml')]
        }
    }
}

This makes my Jenkins build fail if coverage is below 50% at package level. So far so good.

But when a build fails due to this, it is user-hostile and hard to see why. The 'Run unit tests' stage is green in Blue Ocean.

Can I make this stage turn red when it fails the build, so it is easier to see what the error is?

score 1 · Answer 1 · answered Jan 16 '20 at 12:19

1

You can set currentBuild.result to FAILURE if publishCoverage is true. currentBuild.displayName and currentBuild.description optional:

post {
    always {
        script {
            def failed = publishCoverage (failUnhealthy: true, 
                        globalThresholds: [[thresholdTarget: 'Package', unhealthyThreshold: 50.0]],
                        adapters: [coberturaAdapter(
                            mergeToOneReport: true, 
                            path: '**/*.cobertura.xml')])
            if (failed) {
                currentBuild.result = 'FAILURE'
                currentBuild.displayName = "${currentBuild.displayName} Coverage"
                currentBuild.description = "Coverage lower than 50%"
            }
        }
    }
}

answered Jan 16 '20 at 12:19

Sers

12,047
2
12
31

This does not work for me. I now have something very similar to yours in my Jenkinsfile. The stage still remains green, even though the job fails. My log contains the following: "Code coverage enforcement failed: Package coverage in Report level 'Cobertura: **/*.cobertura.xml' is lower than 50.00 healthy threshold". The message in the Jenkinsfile is nowhere to be found in the log. :( – Claus Appel Jan 20 '20 at 13:35
Inspired by your suggestion,I tried to echo the return value of `publishCoverage` itself. Turns out that `publishCoverage` returns null, whether `failUnhealthy` is true or false. Hence the `if` block is never reached. – Claus Appel Jan 20 '20 at 14:31

score 1 · Accepted Answer · answered Jan 21 '20 at 08:16

1

Inspired by the answer from Sers and some other Jenkinsfile code I read through, I arrived at this solution, which does what I want:

stage ('Run unit tests') {
    steps {
        powershell "dotnet test -c:Release /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura --no-build --no-restore --logger trx"
    }
    post {
        always {
            step([$class: 'MSTestPublisher'])
            publishCoverage failUnhealthy: true, 
                globalThresholds: [[thresholdTarget: 'Package', unhealthyThreshold: 50.0]],
                adapters: [coberturaAdapter(
                    mergeToOneReport: true, 
                    path: '**/*.cobertura.xml')]
            script {
                if (currentBuild.result == 'FAILURE') {
                    error("Test coverage is too low.")
                }
            }
        }
    }
}

answered Jan 21 '20 at 08:16

Claus Appel

379
1
4
13

is there any way to achieve it without hard-coding the threshold? that is, fail the build only if the coverage dropped between two consecutive builds? – Mr. Apr 13 '20 at 08:44
Maybe you could somehow extract the current coverage to a variable and read back that variable in the next run. I don't know. – Claus Appel Apr 20 '20 at 05:55
Adding the following will track between targets branches and block merges if coverage is going to be reduced - publishCoverage calculateDiffForChangeRequests: true, failUnhealthy: true, failBuildIfCoverageDecreasedInChangeRequest: true, – BigBadOwl Jan 26 '21 at 10:46

How to make Jenkins stage fail and turn red when coverage is too low (using C# and dotnet test)?

2 Answers2