How to make failsafe-plugin fail if @SpringBootTest could not load ApplicationContext

Question

My @SpringBootTests are executed by maven-failsafe-plugin. That's working fine as long as the ApplicationContext could be loaded. If the ApplicationContext could NOT be loaded (e.g. because of a missing bean in the test) Spring is showing something like

2020-11-04 12:01:41 [main] ERROR o.s.b.d.LoggingFailureAnalysisReporter - 

***************************
APPLICATION FAILED TO START
***************************

Description:
...

2020-11-04 12:01:41 [main] ERROR o.s.test.context.TestContextManager - Caught exception while allowing TestExecutionListener [org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener@3e61138c] to prepare test instance [foo.bar.SomeControllerSecurityIT@4af9cb68]
java.lang.IllegalStateException: Failed to load ApplicationContext
    at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:132)

and the test is not counted at all!

Is this default? Is there a way to count this case as failure?

The junit test report is showing

-------------------------------------------------------------------------------
Test set: foo.bar.SomeControllerSecurityIT
-------------------------------------------------------------------------------
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.636 s - in foo.bar.SomeControllerSecurityIT

even though the test class contains six methods that are running green after fixing the cause (missing bean).

Maybe worth mentioning that the six methods are JUnit @TestFactory methods returning a Stream<DynamicTest>.

Edit: It seems that this is the issue in that case. Implementing a "normal" @Test method results in counting this test and because of a missing/failing initialisation (?) failing that test class with 1 error. That causes the whole build to fail.

Sources to reproduce (as I wrote above, it has nothing to do with spring boot):

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>foo.bar</groupId>
    <artifactId>failing-junit-extension</artifactId>
    <version>0.0.1-SNAPSHOT</version>

    <properties>
        <maven.compiler.target>11</maven.compiler.target>
        <maven.compiler.source>11</maven.compiler.source>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>5.7.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>5.7.0</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>3.0.0-M4</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

FailingJunitExtension.java

package foo.bar;

import org.junit.jupiter.api.extension.BeforeAllCallback;
import org.junit.jupiter.api.extension.ExtensionContext;

public class FailingJunitExtension
    implements
        BeforeAllCallback
{
    public void beforeAll( ExtensionContext context )
        throws Exception
    {
        System.err.println( "Extension thrown in beforeAll" );
        throw new RuntimeException();
    }
}

FailingExtensionIT.java

package foo.bar;

import static org.junit.jupiter.api.Assertions.assertTrue;

import java.util.stream.IntStream;
import java.util.stream.Stream;

import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.TestFactory;
import org.junit.jupiter.api.extension.ExtendWith;

@ExtendWith(FailingJunitExtension.class)
class FailingExtensionIT
{
    @TestFactory
    Stream<DynamicTest> dummyTestFactory()
    {
        return IntStream.iterate( 0, n -> n + 2 ).limit( 10 )
                .mapToObj( n -> DynamicTest.dynamicTest( "test" + n, () -> assertTrue( n % 2 == 0 ) ) );
    }
}

calling mvn clean verify results in

[INFO] Running foo.bar.FailingExtensionIT
Extension thrown in beforeAll
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 s - in foo.bar.FailingExtensionIT
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

No tests at all are registered.

Is this an issue of junit or of the failsafe plugin?

Please add the code for your tests and your Failsafe configuration — rieckpil, Nov 11 '20 at 06:07

score 1 · Answer 1 · answered Dec 09 '20 at 14:20

1

Seems that this issue is solved in failsafe-plugin 3.0.0-M5.

answered Dec 09 '20 at 14:20

Alex

115
1
9

How to make failsafe-plugin fail if @SpringBootTest could not load ApplicationContext

1 Answers1