0

Here is an always failing data-driven spock test:

class HelloSpockSpec extends Specification {
  def "test"() {
    expect:
    1 == number

    where:
    number << [2, 3]
  }
}

When this test is ran from maven, as expected, it executes 2 times, all 2 times it fails and then it fails the build:

mvn test
...
[ERROR] Failures: 
[ERROR] HelloSpockSpec.test(Object)
[ERROR]   Run 1: HelloSpockSpec.test:23 Condition not satisfied:

1 == a
  |  |
  |  2
  false

[ERROR]   Run 2: HelloSpockSpec.test:23 Condition not satisfied:

1 == a
  |  |
  |  3
  false

[INFO]   Run 3: PASS
[INFO] 
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

So far so good, except the [INFO] Run 3: PASS line looks suspicious, as there should be only 2 runs.

When the same test is executed with surefire's rerunFailingTestsCount option:

mvn test -Dsurefire.rerunFailingTestsCount=1
[WARNING] Flakes: 
[WARNING] HelloSpockSpec.test(Object)
[ERROR]   Run 1: HelloSpockSpec.test:23 Condition not satisfied:

1 == a
  |  |
  |  2
  false

[ERROR]   Run 2: HelloSpockSpec.test:23 Condition not satisfied:

1 == a
  |  |
  |  3
  false

[INFO]   Run 3: PASS
[ERROR]   Run 4: HelloSpockSpec.test:23 Condition not satisfied:

1 == a
  |  |
  |  2
  false

[ERROR]   Run 5: HelloSpockSpec.test:23 Condition not satisfied:

1 == a
  |  |
  |  3
  false

[INFO]   Run 6: PASS
[INFO] 
[INFO] 
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

something strange happens:

  • Build is successful, however there is a test which always fails;
  • This test is marked as flaky, meaning it failed and then passed, but again, this test should always fail;
  • The number of Runs is 6, but it should be 4 (2 runs because of where block, multiplied by the number of reruns+1).

The problem seems to come from the Run N: PASS line, it seems that surefire thinks that the test was successful and marks it as flaky.

This issue looks very concerning - combination of data-driven tests and surefire's rerunFailingTestsCount option means that tests might be marked as flaky (even though they always failing), and due to fact that surefire does not fail the maven build, might be unnoticed.

Or am I missing something here?

Here is a github project, check out the build logs

streetturtle
  • 5,472
  • 2
  • 25
  • 43
  • The 3rd run is explained by the fact that the feature itself, i.e., the one that creates the data providers, is a `ContainerAndTest`. Otherwise, Spock couldn't report errors that happened on that level. So you have iteration-1, iteration-2, feature as "runs". – Leonard Brünings Dec 22 '21 at 21:42
  • As for the issue flaky test detection issue, I'm pretty sure that it is a surefire bug. If you didn't configure the correct reporter each run will be reported as ``. When the `JUnit5Xml30StatelessReporter` is used it correctly reports the names as `test [2, #0]` and so on, but I guess that it is not used for the internal matching. JUnit5 introduced the concept of `UniqueId` to avoid that exact problem. Please open an issue on the surefire bugtracker. – Leonard Brünings Dec 22 '21 at 21:48
  • Thanks! I'll raise an issue there. I've added link to a simple project which reproduces the issue, just in case somebody wants to see it in action. – streetturtle Dec 23 '21 at 02:38
  • I've created an issue in Surefire's jira: https://issues.apache.org/jira/browse/SUREFIRE-1970 – streetturtle Dec 23 '21 at 03:13

0 Answers0