order execution of Junit5 tests by launcher / engine

Question

I have a Spring Boot 2.5.4 project with some @SpringBootTest tests and some @Cucumber tests. I am using gradle to build.

I have noticed that my build is failing depending on where it's executed, and I found that it actually depended on the order the tests get executed, so I have a problem in my tests : if @SpringBootTest runs first then it's passing. if @Cucumber tests run first then it fails - probably because the H2 DB doesn't get fully reset in between.

Now, I would like to temporarily control the execution order, so that I can reproduce the issue consistently to fix the data dependency between my tests.

I am trying to use junit.jupiter.testclass.order.default property with value org.junit.jupiter.api.ClassOrderer$ClassName but it's not working.

I've put my 2 tests in a Juint5 @Suite mentioning the 2 tests in the @SelectClasses and changing their order, but even like that, it's not working - my feeling is that it's because there are actually 2 test runners, Junit Jupiter and Cucumber. Sometimes when I change something that doesn't seem related, the execution order changes :

I'm overriding Junit version to latest, hoping that it helps (and Junit5 Suite is available), but it doesn't help :

ext['junit-jupiter.version']='5.9.2'

I am using Cucumber 6.11.0.

My gradle test task is simply

test {
    useJUnitPlatform()
    finalizedBy(project.tasks.jacocoTestReport)
}

Is there a way to configure in my build the order in which the test runners get executed, in a consistent and reproduceable manner ?

Thanks

If you have two separate suites, can you get Gradle to run them in alphabetical order? — M.P. Korstanje, Feb 17 '23 at 22:38
junit.jupiter.testclass.order.default is only for the Jupiter engine. In your case, however, three engines are involved: Jupiter, Cucumber and Suite. — johanneslink, Feb 18 '23 at 06:20

score 1 · Accepted Answer · edited Feb 18 '23 at 08:28

Gradle uses the JUnit Platform Launcher API to run the tests. You can do the same thing. And to ensure the order remains the same, you would invoke the platform twice within the same JVM.

This should allow you to reproduce your problem as Spring keeps the application running until the JVM exits.

import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass;

import my.project.spring.MySpringBootTest;
import my.project.cucumber.MyCucumberTest;
import org.junit.platform.launcher.Launcher;
import org.junit.platform.launcher.LauncherDiscoveryRequest;
import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder;
import org.junit.platform.launcher.core.LauncherFactory;
import org.junit.platform.launcher.listeners.SummaryGeneratingListener;
import org.junit.platform.launcher.listeners.TestExecutionSummary;

public class ManualLaunch {

  public static void main(String[] args) {

    var cucumberSummary= runCucumberTest();
    var jupiterSummary= runJupiterTest();
    
    System.out.println("Jupiter failures : "+jupiterSummary.getTestsFailedCount());
    System.out.println("Cucumber failures : "+cucumberSummary.getTestsFailedCount());

  }

  private static TestExecutionSummary runCucumberTest() {

    LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
        // Configure the request for Cucumber here
        .selectors(
            selectClass(MyCucumberTest.class)
        )
        .build();

    return launchAndGetSummary(request);
  }

    private static TestExecutionSummary runJupiterTest() {
      LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
          .selectors(
              selectClass(MySpringBootTest.class)
          )
          .build();
      
      return launchAndGetSummary(request);
    }
    
    private static TestExecutionSummary launchAndGetSummary(LauncherDiscoveryRequest request){
      Launcher launcher = LauncherFactory.create();
      SummaryGeneratingListener listener = new SummaryGeneratingListener();
      launcher.registerTestExecutionListeners(listener);
      launcher.execute(request);

      return listener.getSummary();
      
    }

}

If you don't know how to run a main method from Gradle, you could also use JUnit to run JUnit. But then you do have to make sure the tests don't target themselves.

Thanks, I've updated your code with what I ended up with - it gets the job done nicely for investigating ! — Vincent F, Feb 18 '23 at 08:29

order execution of Junit5 tests by launcher / engine

1 Answers1