2

I'm using Scalatest/Scalacheck with custom generators. I observe that tests are marked as success even if some tests failed. In below example test "should add processing timestamp" was Falsified. Yet sbt test passed.

+ OK, passed 100 tests.
[info] - should add product info to event 
[info] - should not alter rest of event
+ OK, passed 100 tests.
! Falsified after 0 passed tests.
> ARG_0: List("([B@27d10fe1,...)")
> ARG_0_ORIGINAL: List("([B@3c8057ce,...)")
[info] - should add processing timestamp
[info] ScalaTest
[info] Run completed in 4 seconds, 792 milliseconds.
[info] Total number of tests run: 3
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 3, Failed 0, Errors 0, Passed 3
[success] Total time: 8 s, completed Sep 11, 2017 6:54:28 PM

Why is test not failing??

UPDATE: sbt 0.13, scalatest 3.0.1, scalacheck 1.13.4 and the test case is

  it should "add processing timestamp" in {
    Prop.forAll(sizedGen(TestInputGen.Generate)) { in => 
      val out = processor.input(in)

      out.forall(o => {
        val outTS = o._2.get("timestamps")
        (outTS.getModule() == "PrimaryProcessor")
      })
    }
  }.check
Cheeko
  • 1,193
  • 1
  • 12
  • 23
  • The test is failing. The task is succeding. – pedromss Sep 12 '17 at 20:50
  • """[info] Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0""" --- Failed test is 0. – Cheeko Sep 13 '17 at 04:05
  • Ok. Can you provide the test that is failing but passing? The command you used to run (sbt test i assume) and the versions of sbt and scalatest. – pedromss Sep 13 '17 at 07:14
  • I've updated the post with the info you asked. Yes, I run sbt test or testOnly. – Cheeko Sep 14 '17 at 15:11
  • Possible duplicate of [How to use scalacheck prop generators in scalatest FlatSpec](https://stackoverflow.com/questions/29059484/how-to-use-scalacheck-prop-generators-in-scalatest-flatspec) – Cheeko Oct 12 '17 at 14:09

1 Answers1

0

Since you are using ScalaTest style of property based testing as opposed to the ScalaCheck style of property testing, your properties need to return a Matcher or an Assertion instead of a Boolean expression.

From the documentation:

In the ScalaTest property style you use the word whenever instead of ==> and either an assertion or matcher expression instead of a boolean expression

Below in an example you can use to test this. The test tests that the length of 2 concatenated strings is always bigger than the length of any of the strings used in the concatenation. This should fail when both strings are empty

Compiles but tests pass because we are using a Boolean expression

import org.scalatest.prop.PropertyChecks
import org.scalatest.{FlatSpec, Matchers}

final class StringSpec extends FlatSpec with Matchers with PropertyChecks {

  behavior of "String"

  it should "concatenate" in {
    forAll { (a: String, b: String) =>
      (a + b).length > a.length && (a + b).length > b.length
    }
  }
}

Compiles and fails as expected because it uses `Matchers

import org.scalatest.prop.PropertyChecks
import org.scalatest.{Assertion, FlatSpec, Matchers}

final class StringSpec extends FlatSpec with Matchers with PropertyChecks {

  behavior of "String"

  it should "concatenate" in {
    forAll { (a: String, b: String) =>
      (a + b).length should be > a.length
      (a + b).length should be >= b.length
    }
  }
}
pedromss
  • 2,443
  • 18
  • 24
  • Thanks for your response. While this is in the right direction, I found that a more accurate answer was already given in another SO question. I've marked this as a duplicate of the other SO question. – Cheeko Oct 12 '17 at 14:11
  • https://stackoverflow.com/questions/29059484/how-to-use-scalacheck-prop-generators-in-scalatest-flatspec – Cheeko Oct 12 '17 at 14:12