I have some Cucumber-Java test suites in some of my Java projects. They're set up with JUnit 4 runners using the @RunWith annotation and the Cucumber runner.
I just upgraded from IntelliJ 18.2 to 18.3 (Ultimate). In 18.2, the unit test would fail if there was a failed assertion; in 18.3, the failed step is showing as "skipped" and the test is passing. Is there a configuration change I need to make in order to get this to work? I'm looking for a solution that will allow me to treat both fails and skips as failures, since that seems to be the most robust solution which most closely mimics previous behavior.
The runner looks like this:
@RunWith(Cucumber.class)
@CucumberOptions(plugin = {"junit:target/junitTests-example.xml"},
monochrome = true,
glue = {"path.to.glue"},
features = "src/test/resources/features/")
public class MyTest { }
The glue looks like this (that's a JUnit 4 assertion):
@When("^I run a failing test$")
public void runTrivialFailureTest() {
assertNull("This test should always fail.");
}
And the scenario definition is like:
Feature: Sample
Scenario: A trivial failing test
Given I run a failing test
Scenario: A trivial test with an undefined step
Given I run a test with an undefined step
In 2018.2 IntelliJ, this test would have been shown as a failing unit test. In 2018.3, the failing step is showed as skipped and the unit test passes. There is no difference in behavior if the test class is run as a JUnit test, or if the scenario is run using the Cucumber java plugin.
The project is a Maven project, and from the command-line, when invoking the surefire plugin, it correctly shows the tests as failures. It's only when running within IntelliJ that these are treated as "skips".
