cucumber.json report getting overwritten by rerun scenario report

Question

I have got UI Test project and a API test project with same technology stack (JAVA1.8, Cucumber-JVM, JUnit, Maven) and both projects are showing me this problem. Probably because same set of dependencies are present in both.

I have employed the Flaky test re-run mechanism using maven-surefire-plugin build-in functionality <rerunFailingTestsCount>1</rerunFailingTestsCount>. Also, I have cucumber dependencies added based on <groupId>io.cucumber</groupId> and not <groupId>info.cukes</groupId>. Both these have their own version of cucumber-java and cucumber-jvm dependencies.

My POM.XML looks like this.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.21.0</version>
    <configuration>
         <rerunFailingTestsCount>1</rerunFailingTestsCount>
    </configuration>
    <executions>
         <execution>
             <id>acceptance-test</id>
             <phase>integration-test</phase>
             <goals>
                 <goal>test</goal>
             </goals>
             <configuration>
                 <forkCount>1</forkCount>
                 <reuseForks>true</reuseForks>
                  <includes>
                      <include>**/*Runner.class</include>
                  </includes>
              </configuration>
         </execution>
    </executions>
</plugin>
<dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-java</artifactId>
        <version>2.4.0</version>
    </dependency>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-junit</artifactId>
        <version>2.4.0</version>
    </dependency>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-spring</artifactId>
        <version>2.4.0</version>
    </dependency>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-jvm</artifactId>
        <version>2.4.0</version>
        <type>pom</type>
    </dependency>

ONLY RUNNER FILE CODE

@RunWith(Cucumber.class)
@ContextConfiguration(locations = {"file:/src/test/resources/spring-config.xml"})
@CucumberOptions(
        glue = "com.test.uitest",
        features = "classpath:cucumber",
        tags = {"~@ignore","@ui_home"},
        monochrome = true,
        plugin = {"pretty", "html:target/cucumber-reports",
                "json:target/cucumber-reports/cucumber.json",
                "rerun:target/rerun.txt"} //Creates a text file with failed scenarios
)
public class AllTestsRunner {
}

Now Aparently, I need to have another runner with following code in it (as per other forums and threads here on StackOverflow)

@RunWith(Cucumber.class)
@CucumberOptions(
        monochrome = true,
        glue = "com.test.uitest",
        features = "@target/rerun.txt", //Cucumber picks the failed scenarios from this file
        format = {"pretty", "html:target/rerun-reports",
                "json:target/cucumber-reports/rerun-report.json"}
)
public class FailedTestsRunner {
}

But I don't need to have this 2nd runner, as the rerun mechanism works absolutely brilliantly with just the one runner on top. Even, there is no need of rerun.txt file generated in 1st runner.The in-build mechanism within maven-surefire plugin (v_2.21.0) along with io.cucumber v_2.4.0 works perfectly and if any scenario fails during 1st execution, it reruns automatically without recording it inside rerun.txt file.

## PROBLEM IS ##

I have 5 scenarios in my feature file. If all of them pass in 1st run. It successfully generates the report with cucumber.json report showing all 5 scenarios. But, if (say) 2 out of 5 scenarios fails, and those gets executed automatically in a rerun mechanism, the cucumber.json report file only recording results of those two scenarios and not the all 5 scenarios. Overall build PASSES if those 2 scenarios passes in rerun or FAILs if those 2 scenarios fails. That's correct, but my problem is with the cucumber.json getting overwritten by rerun mechanism. I have tried to use the maven-cucumber-reporting plugin v_3.16.0 but it actually reads the cucumber.json file itself and hence don't resolve my problem. Any help would be appreciated.

Answer 1

The good news is you're not doing anything wrong!

The bad news is that the results you're observing are entirely as expected. This is a natural consequence of chaining different tools (Surefire --> JUnit --> Cucumber) that are otherwise unaware of each other. From the perspective of Cucumber it would appear that the rerun is an entirely new execution so it will happily overwrite the old reports. Only at the start of the chain it is possible to create accurate reports.

As such your options from least to most effort and worst to best quality are:

1. Use the reports generated by Surefire.

2. Write your own plugin that appends results rather then overwriting them.

3. Get involved with the Cucumber project and help resolve this fundamentally.

Edit: Removed the suggestion to use the cucumber-jvm-parallel-plugin create a individual unit test for each scenario. This won't actually work.

Answer 2

In this case the AllTestsRunner is run repeatedly till it succeeds up to the count mentioned in the rerunFailingTestsCount parameter or fails on last run. So the last run always wins.

The surefire plugin creates its own reports in the target folder. Two reports are AllTestsRunner.txt, which is a summary, and TEST-AllTestsRunner.xml has the details. Both these reports has details about all runs. You could create a custom program to convert the TEST-AllTestsRunner.xml file to desired json.

There is a surefire report plugin which reads the above xml file and generates html report. It will create reports in the site folder and run with mvn site. Maybe this works.

Answer 3

I face the same issue. I'm pretty happy with the result I got. Share the info here.

Retry without concern of report

If you just want to retry the failed tests and doesn't care about the incomplete report, it's easy. There are two ways to do it.

1. Use rerunFailingTestsCount from Maven. Reference https://maven.apache.org/surefire/maven-failsafe-plugin/examples/rerun-failing-tests.html. Regarding reporting, the limitation is cucumber.json is overridden during the rerun. That means you will only get the report of tests which have been rerun.
2. Use rerun plugin of Cucumber. See the code given by N Riay. Regarding reporting, the limitation is there are two cucumber json files generated.

Solution

If you also want to get an unified report, here is the solution. On top of option 2, we use cucumber-reporting to combine the two cucumber json files after all the runners including FailedTestsRunner are finished.

Please refer to code in section "ONLY RUNNER FILE CODE" given by N Riay. Then, add below code. BeforeSuite and AfterSuite are concept of TestNG. For Junit, it's similar. Just to make sure it's only run once.

import net.masterthought.cucumber.Configuration;
import net.masterthought.cucumber.ReportBuilder;
import net.masterthought.cucumber.reducers.ReducingMethod;

@BeforeSuite
public void beforeSuite() throws IOException {
    <Delete all the cucumber json files. Otherwise report will always combine json file of FailedTestsRunner even it's not run>
}

@AfterSuite
public void afterSuite() {
    Configuration configuration = new Configuration(new File("target"), "my project");
    configuration.addReducingMethod(ReducingMethod.MERGE_FEATURES_WITH_RETEST);
    List<String> jsonFiles = <get the file path of the two json>;
    ReportBuilder reportBuilder=new ReportBuilder(jsonFiles,configuration);
    reportBuilder.generateReports();
}

MERGE_FEATURES_WITH_RETEST reference - https://javadoc.io/static/net.masterthought/cucumber-reporting/5.0.2/net/masterthought/cucumber/reducers/ReducingMethod.html

Answer 4

Edited: I tried with solution provided by gary.zhang by creating method in hooks class.but still not able to get consolidated report after rerun. Only latest failed scenario is generated