Quality Gate does not fail when conditions for success are not met

Question

I have established Quality Gate for my Jenkins project via SonarQube. One of my projects have no tests at all, so in the analysis I see that the code coverage is 0%. By the quality gate rules (<60% coverage = fail) my pipeline should return an error. However, this does not happen. The quality gate says that the analysis was a success and quality gate is 'OK'. In another project, I removed some tests to make coverage be <60% and the quality gate passed once again, even though it was meant to fail.

I had an error related to the analysis always returning 0% coverage before, but managed to fix it (with help from this link). Found a lot of articles with the similar questions but with no answers on any of them. This post looks promising but I cannot find the suitable alternative to its suggestion.

It is worth mentioning that the analysis stage is done in parallel with another stage (to save some time). The Quality Gate stage comes shortly afterwards.

The relevant code I use to initialise the analysis for my project is (org.jacoco... bit is the solution to 0% coverage error I mentioned above):

sh "mvn clean org.jacoco:jacoco-maven-plugin:prepare-agent verify sonar:sonar -Dsonar.host.url=${env.SONAR_HOST_URL} -Dsonar.login=${env.SONAR_AUTH_TOKEN} -Dsonar.projectKey=${projectName} -Dsonar.projectName=${projectName} -Dsonar.sources=. -Dsonar.java.binaries=**/* -Dsonar.language=java -Dsonar.exclusions=$PROJECT_DIR/src/test/java/** -f ./$PROJECT_DIR/pom.xml"

The full quality gate code (to clarify how my quality gate starts and finishes):

 stage("Quality Gate") {
      steps {
        timeout(time: 15, unit: 'MINUTES') { // If analysis takes longer than indicated time, then build will be aborted
            withSonarQubeEnv('ResearchTech SonarQube'){
                script{

                // Workaround code, since we cannot have a global webhook
                    def reportFilePath = "target/sonar/report-task.txt"
                    def reportTaskFileExists = fileExists "${reportFilePath}"

                    if (reportTaskFileExists) {
                        def taskProps = readProperties file: "${reportFilePath}"
                        
                        def authString = "${env.SONAR_AUTH_TOKEN}"                 
                        def taskStatusResult    =
                            sh(script: "curl -s -X GET -u ${authString} '${taskProps['ceTaskUrl']}'", returnStdout: true)
                            //echo "taskStatusResult[${taskStatusResult}]"
                        def taskStatus  = new groovy.json.JsonSlurper().parseText(taskStatusResult).task.status
                        echo "taskStatus[${taskStatus}]"

                        if (taskStatus == "SUCCESS") {
                            echo "Background tasks are completed"
                        } else {
                            while (true) {
                                sleep 10
                                taskStatusResult    =
                                    sh(script: "curl -s -X GET -u ${authString} '${taskProps['ceTaskUrl']}'", returnStdout: true)
                                    //echo "taskStatusResult[${taskStatusResult}]"
                                taskStatus  = new groovy.json.JsonSlurper().parseText(taskStatusResult).task.status
                                echo "taskStatus[${taskStatus}]"
                                if (taskStatus != "IN_PROGRESS" && taskStatus != "PENDING") {
                                    break;
                                }
                            }
                        }
                    } else {
                        error "Haven't found report-task.txt."
                    }

                    def qg = waitForQualityGate() // Waiting for analysis to be completed
                    if(qg.status != 'OK'){ // If quality gate was not met, then present error
                        error "Pipeline aborted due to quality gate failure: ${qg.status}"
                    }
                }
            }
        }
      }
    }

Answer 1

What is shown in the SonarQube UI for the project? Does it show that the quality gate failed, or not?

I don't quite understand what you're doing in that pipeline script. It sure looks like you're calling "waitForQualityGate()" twice, but only checking for error on the second call. I use scripted pipeline, so I know it would look slightly different.

Update:

Based on your additional comment, if the SonarQube UI says that it passed the quality gate, then that means there's nothing wrong with your pipeline code (at least with respect to the quality gate). The problem will be in the definition of your quality gate.

However, I would also point out one other error in how you're checking for the background task results.

The possible values of "taskStatus" are "SUCCESS", "ERROR", "PENDING", and "IN_PROGRESS". If you need to determine whether the task is still running, you have to check for either of the last two values. If you need to determine whether the task is complete, you need to check for either of the first two values. You're checking for completion, but you're only checking for "SUCCESS". That means if the task failed, which it would if the quality gate failed (which isn't happening here), you would continue to wait for it until you timed out.