Forking a JVM process per feature file?

Question

I have a number of feature files in my cucumber scenario test suite.

I run the tests by launching Cucumber using the CLI.

These are the steps which occur when the test process is running:

• We create a static instance of a class which manages the lifecycle of testcontainers for my cucumber tests.
  • This currently involves three containers: (i) Postgres DB (with our schema applied), (ii) Axon Server (event store), (iii) a separate application container.
  • We use spring's new @DynamicPropertySource to set the values of our data source, event store, etc. so that the cucumber process can connect to the containers.
• @Before each scenario we perform some clean up on the testcontainers.
  • This is so that each scenario has a clean slate.
  • It involves truncating data in tables (postgres container), resetting all events in our event store (Axon Server container), and some other work for our application (resetting relevant tracking event processors), etc.

Although the tests pass fine, the problem is by default it takes far too long for the test suite to run. So I am looking for a way to increase parallelism to speed it up.

• Adding the arguments --threads <n> will not work because the static containers will be in contention (and I have tried this and as expected it fails).

The way I see it there is are different options for parallelism which would work:

1. Each scenario launches its own spring application context (essentially forking a JVM), gets its own containers deployed and runs tests that way.
2. Each feature file launches its own spring application context (essetially forking a JVM), gets its own containers deployed and runs each scenario serially (as it would normally).

I think in an ideal world we would go for 1 (see *). But this would require a machine with a lot of memory and CPUs (which I do not have access to). And so option 2 would probably make the most sense for me.

My questions are:

• is it possible to configure cucumber to fork JVMs which run assigned feature files (which matches option 2 above?)
• what is the best way to parallelise this situation (with testcontainers)?

* Having each scenario deployed and tested independently agrees with the cucumber docs which state: "Each scenario should be independent; you should be able to run them in any order or in parallel without one scenario interfering with another. Each scenario should test exactly one thing so that when it fails, it fails for a clear reason. This means you wouldn’t reuse one scenario inside another scenario."

Answer 1

This isn't really a question for stack overflow. There isn't a single correct answer - mostly it depends. You may want to try https://softwareengineering.stackexchange.com/ in the future.

No. This is not possible. Cucumber does not support forking the JVM. Surefire however does support forking and you may be able to utilize this by creating a runner for each feature file.

However I would reconsider the testing strategy and possibly the application design too. To execute tests in parallel your system has to support parallel invocations. So I would not consider resetting your database and event store for each test a good practice. Instead consider writing your tests in such a way that each test uses its own isolated set of resources. So for example if you are testing users, you create randomized users for each test. If these users are part of an organization, you create a random organization, ect. This isn't always possible. Some applications are designed with implicit singleton resources in the code. In this case you'll have to refactor the application to make these resources explicit.

Alternatively consider pushing your Cucumber tests down the stack. You can test business logic at any abstraction level. It doesn't have to be an integration test. Then you can use JUnit with Surefire instead and use Surefire to create multiple forks.