Maven surefire plugin outputs all errors at the end - causing OOME on large tests

Question

Background:

We have a regression test suite that tests the generation of some large xml files by comparing them field-by-field to the corresponding baseline files. This is implemented using a junit4 parameterized test running the test for each file and assertj soft assertions to collect the field comparison errors.

Problem:

When I run this test from my IDE, I can see the assertion errors output after each test (each file), but when run from maven, surefire collects all the errors in memory and outputs them at the end (when all the tests for the class have finished). Now, running this for 2000+ files, comparing hundreds of fields in each and having a lot of differences results in OutOfMemoryError, even with 8GB of heap allocated.

Question:

I'm trying to find out if there's any option in surefire to either output the errors after each test or not collect & output them at all (we're logging them into a file and generating custom reports from there anyway).

I've tried <redirectTestOutputToFile>true</redirectTestOutputToFile> but this only redirects stdout (the logs produced during test execution), the assertion errors are still spit to console after the tests finish.

Other options I see are:

1. Split the test parameters into smaller batches and run the test suite for each batch, then aggregate the results - this could be done in the Jenkins job.
2. Remove the detailed error reporting using soft assertions and only have a single assertion at the end of the test. This is what we had before and obviously didn't help in finding errors. I wouldn't like to go back there.
3. Add an option to run the tests in two modes:
   • use soft assertions to provide detailed error reporting when run locally (with a smaller set of data)
   • use a single assertion when run on Jenkins (with the full set of data) - here we're only interested in the logs, not the console output

The third solution would result in some ifs in the code and would make the code less readable, that's why I'm trying to solve this from configuration first.

Any other ideas? Thanx!