I have set up Jenkins test jobs that will run Cucumber tests and want my jobs to always be successful up until a certain threshold (which I am controlling). The Allure plugin is always marking my builds as "UNSTABLE" when I have failed tests but I do not want my builds to be unstable. How can I change this behavior with allure?
I have tried setting
<testFailureIgnore>true</testFailureIgnore>
in pom.xml which is allowing the Maven Build to be successful. However, I am manually checking the Allure results in prometheusData.txt that is generated and failing/passing builds based off that result.
I want to be able to set Success/Failure status based off the 80% threshold. I am not aware of any thresholding that exists with Allure right now which is why I had to do it this way
Pipeline script
def testsInfo = readFile "${env.WORKSPACE}/allure-report/export/prometheusData.txt"
def passingTests = 0;
def totalTests = 0
def lineText = ''
def lineNum = 0
while(lineText != 'found'){
findText = testsInfo.split("\n")[lineNum]
if(findText.contains('launch_status_passed')){
passingTests = findText.split(' ')[1].toInteger()
}
if(findText.contains('launch_retries_run')){
totalTests = findText.split(' ')[1].toInteger()
lineText='found'
}
lineNum++
}
echo 'passing tests: ' + passingTests + ' total tests: ' + totalTests
percentSuccess = ((passingTests / totalTests) * 100).toString().split('\\.')[0]
echo 'Percent of tests that passed: ' + percentSuccess
if(build_ok && (percentSuccess.toInteger() > 80)) {
currentBuild.result = 'SUCCESS'
echo 'build OK'
} else {
currentBuild.result = 'FAILURE'
echo 'build bad'
}
Jenkins output
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] expected:<true> but was:<false>
[ERROR] expected:<[1].00> but was:<[2].00>
[ERROR] expected:<1.[0]0> but was:<1.[4]0>
[INFO]
[ERROR] Tests run: 90, Failures: 3, Errors: 0, Skipped: 0
[INFO]
[ERROR] There are test failures.
[INFO] -----------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] -----------------------------------------------------------------
[INFO] Total time: 04:05 min
[INFO] Finished at: 2019-06-24T17:31:36Z
[INFO] Final Memory: 33M/1349M
[INFO] -----------------------------------------------------------------
Allure report was successfully generated.
Creating artifact for the build.
Artifact was added to the build.
[Pipeline] // stage
[Pipeline] readFile
[Pipeline] echo
passing tests: 96 total tests: 100
[Pipeline] echo
Percent of tests that passed: 96
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: UNSTABLE