3

I have set up Jenkins test jobs that will run Cucumber tests and want my jobs to always be successful up until a certain threshold (which I am controlling). The Allure plugin is always marking my builds as "UNSTABLE" when I have failed tests but I do not want my builds to be unstable. How can I change this behavior with allure?

I have tried setting <testFailureIgnore>true</testFailureIgnore> in pom.xml which is allowing the Maven Build to be successful. However, I am manually checking the Allure results in prometheusData.txt that is generated and failing/passing builds based off that result.

I want to be able to set Success/Failure status based off the 80% threshold. I am not aware of any thresholding that exists with Allure right now which is why I had to do it this way

Pipeline script

def testsInfo = readFile "${env.WORKSPACE}/allure-report/export/prometheusData.txt"

def passingTests = 0;
def totalTests = 0 
def lineText = ''
def lineNum = 0
while(lineText != 'found'){
    findText = testsInfo.split("\n")[lineNum]
    if(findText.contains('launch_status_passed')){
        passingTests = findText.split(' ')[1].toInteger()
    }
    if(findText.contains('launch_retries_run')){
        totalTests = findText.split(' ')[1].toInteger()
        lineText='found'
    }
    lineNum++
}

echo 'passing tests: ' + passingTests + '    total tests: ' + totalTests
percentSuccess = ((passingTests / totalTests) * 100).toString().split('\\.')[0]
echo 'Percent of tests that passed: ' + percentSuccess


if(build_ok && (percentSuccess.toInteger() > 80)) {
    currentBuild.result = 'SUCCESS'
    echo 'build OK'
} else {
    currentBuild.result = 'FAILURE'
    echo 'build bad'
}

Jenkins output

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   expected:<true> but was:<false>
[ERROR]   expected:<[1].00> but was:<[2].00>
[ERROR]   expected:<1.[0]0> but was:<1.[4]0>
[INFO] 
[ERROR] Tests run: 90, Failures: 3, Errors: 0, Skipped: 0
[INFO] 
[ERROR] There are test failures.
[INFO] -----------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] -----------------------------------------------------------------
[INFO] Total time: 04:05 min
[INFO] Finished at: 2019-06-24T17:31:36Z
[INFO] Final Memory: 33M/1349M
[INFO] -----------------------------------------------------------------
Allure report was successfully generated.
Creating artifact for the build.
Artifact was added to the build.
[Pipeline] // stage
[Pipeline] readFile
[Pipeline] echo
passing tests: 96    total tests: 100
[Pipeline] echo
Percent of tests that passed: 96
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: UNSTABLE
Vadim Kotov
  • 8,084
  • 8
  • 48
  • 62
samkul
  • 31
  • 3
  • Have you checked the [definitions](https://wiki.jenkins.io/plugins/servlet/mobile?contentId=3604507)? A build can be successful/failed based on the build, stable/unstable if all tests pass or some fail. I believe if less than threshold pass, the it's alao considered a fail. Not familiar w/Allure. – Ian W Jun 29 '19 at 03:30
  • show us the rest of you pipeline code please. – hakamairi Jul 09 '19 at 07:09

0 Answers0