How to change Jenkins build status if failed scenarios were successfully reruned

Question

I have two TestNG runners to run main and failed scopes of cucumber scenarios. In case MainTestRunner was successfull, Jenkins build status is "success" and FailedTestRunner will not find any failed scenarios into rerun.txt. But in case MainTestRunner has failed scenario and it was reruned in scope of FailedTestRunner and passed, Jenkins build is failed anyway.

I need a solution to make build status "green" as all tests are passed in the end.

There is no special tricks in my test runners:

MainTestRunner:

@CucumberOptions(
        features = ".",
        glue = {"steps"},
        monochrome = true,
        format = {
                "pretty",
                "html:target/cucumber-pretty",
                "json:target/CucumberTestReport.json",
                "rerun:target/rerun.txt"
        })

public class MainTestRunner extends AbstractTestNGCucumberTests {
private TestNGCucumberRunner testNGCucumberRunner;

@BeforeClass(alwaysRun = true)
public void setUpClass() {
    testNGCucumberRunner = new TestNGCucumberRunner(this.getClass());
}

@Test(description = "Runs Cucumber Feature", dataProvider = "features")
public void feature(CucumberFeatureWrapper cucumberFeature) {
    testNGCucumberRunner.runCucumber(cucumberFeature.getCucumberFeature());
}

@DataProvider
public Object[][] features() {
    return testNGCucumberRunner.provideFeatures();
}

@AfterClass(alwaysRun = true)
public void tearDownClass() {
    testNGCucumberRunner.finish();
}
}

FailedTestRunner:

@CucumberOptions(
    features = "@target/rerun.txt"
    , glue = {"steps"}
    , monochrome = true
    , format = {
    "pretty",
    "html:target/cucumber-pretty",
    "json:target/CucumberTestReport.json",
    "rerun:target/rerun.txt"
})

public class FailedTestRunner extends AbstractTestNGCucumberTests {}

You might want to run it as a shell script. Do set e then mvn test then read data of failures somehow, maybe have a failedcount written in a listener and read in the script and then do exit accordingly. — niharika_neo, Jan 09 '19 at 16:37

score 0 · Answer 1 · answered Jan 12 '19 at 03:17

If I could see your pipeline steps it would help the answer, but If you execute your cucumber tests through sh step, then you are able to get the exit status of you executions with returnStatus:true. If main fails and sh returns exit status > 0 then the you can run the failed without changing the build status. If the replay fails, then you let it return a failed status if main passes then you can just skip the failed.

score 0 · Accepted Answer · answered Jan 17 '19 at 13:44

The main goal was to get 'green' Jenkins status if second test run was successful.
I added one more build step. Now I'm using:

mvn clean test -Dmaven.test.failure.ignore=true

to record all failed tests to rerun.txt file.
And:

mvn test -Dtest=runners.FailedTestRunner -DfailIfNoTests=false

to run it.

How to change Jenkins build status if failed scenarios were successfully reruned

2 Answers2