3

I'm trying to replace some old unit tests with property based testing (PBT), concreteley with scala and scalatest - scalacheck but I think the problem is more general. The simplified situation is , if I have a method I want to test: 

 def upcaseReverse(s:String) = s.toUpperCase.reverse

Normally, I would have written unit tests like: 

assertEquals("GNIRTS", upcaseReverse("string"))
assertEquals("", upcaseReverse(""))
// ... corner cases I could think of

So, for each test, I write the output I expect, no problem. Now, with PBT, it'd be like : 

property("strings are reversed and upper-cased") {
 forAll { (s: String) =>
   assert ( upcaseReverse(s) == ???) //this is the problem right here!
 }
}

As I try to write a test that will be true for all String inputs, I find my self having to write the logic of the method again in the tests. In this case the test would look like :

   assert ( upcaseReverse(s) == s.toUpperCase.reverse)

That is, I had to write the implementation in the test to make sure the output is correct. Is there a way out of this? Am I misunderstanding PBT, and should I be testing other properties instead, like :

  • "strings should have the same length as the original"
  • "strings should contain all the characters of the original"
  • "strings should not contain lower case characters" ...

That is also plausible but sounds like much contrived and less clear. Can anybody with more experience in PBT shed some light here?

EDIT : following @Eric's sources I got to this post, and there's exactly an example of what I mean (at Applying the categories one more time): to test the method times in (F#): 

type Dollar(amount:int) =
member val Amount  = amount 
member this.Add add = 
    Dollar (amount + add)
member this.Times multiplier  = 
    Dollar (amount * multiplier)
static member Create amount  = 
    Dollar amount

the author ends up writing a test that goes like: 

let ``create then times should be same as times then create`` start multiplier = 
let d0 = Dollar.Create start
let d1 = d0.Times(multiplier)
let d2 = Dollar.Create (start * multiplier)      // This ones duplicates the code of Times!
d1 = d2

So, in order to test that a method, the code of the method is duplicated in the test. In this case something as trivial as multiplying, but I think it extrapolates to more complex cases.

Chirlo
  • 5,989
  • 1
  • 29
  • 45

2 Answers2

5

This presentation gives some clues about the kind of properties you can write for your code without duplicating it.

In general it is useful to think about what happens when you compose the method you want to test with other methods on that class:

  • size
  • ++
  • reverse
  • toUpperCase
  • contains

For example:

  • upcaseReverse(y) ++ upcaseReverse(x) == upcaseReverse(x ++ y)

Then think about what would break if the implementation was broken. Would the property fail if:

  1. size was not preserved?
  2. not all characters were uppercased?
  3. the string was not properly reversed?

1. is actually implied by 3. and I think that the property above would break for 3. However it would not break for 2 (if there was no uppercasing at all for example). Can we enhance it? What about:

  • upcaseReverse(y) ++ x.reverse.toUpper == upcaseReverse(x ++ y)

I think this one is ok but don't believe me and run the tests!

Anyway I hope you get the idea:

  1. compose with other methods
  2. see if there are equalities which seem to hold (things like "round-tripping" or "idempotency" or "model-checking" in the presentation)
  3. check if your property will break when the code is wrong

Note that 1. and 2. are implemented by a library named QuickSpec and 3. is "mutation testing".

Addendum

About your Edit: the Times operation is just a wrapper around * so there's not much to test. However in a more complex case you might want to check that the operation:

  • has a unit element
  • is associative
  • is commutative
  • is distributive with the addition

If any of these properties fails, this would be a big surprise. If you encode those properties as generic properties for any binary relation T x T -> T you should be able to reuse them very easily in all sorts of contexts (see the Scalaz Monoid "laws").

Coming back to your upperCaseReverse example I would actually write 2 separate properties:

 "upperCaseReverse must uppercase the string" >> forAll { s: String =>
    upperCaseReverse(s).forall(_.isUpper)
 }

 "upperCaseReverse reverses the string regardless of case" >> forAll { s: String =>
    upperCaseReverse(s).toLowerCase === s.reverse.toLowerCase
 }

This doesn't duplicate the code and states 2 different things which can break if your code is wrong.

In conclusion, I had the same question as you before and felt pretty frustrated about it but after a while I found more and more cases where I was not duplicating my code in properties, especially when I starting thinking about

  • combining the tested function with other functions (.isUpper in the first property)
  • comparing the tested function with a simpler "model" of computation ("reverse regardless of case" in the second property)
Eric
  • 15,494
  • 38
  • 61
  • How I understand your point (and the presentation you link) is that PBT defines properties "around my code", like : it commutes or not, it composes with this other methods... but these properties don't really express what the function's result should be. If I want to have a property that says "upcaseReverse will reverse and upcase any String", I don't see any other way to test it than duplicating the code. The property you propose (`upcaseReverse(y) ++ x.reverse.toUpper == upcaseReverse(x ++ y)`) is already somehow duplicating the code, just calling the methods in a different order. – Chirlo Jun 25 '15 at 13:13
  • Noooo, I had an additional edit which was lost apparently. I'll add the gist of it back. – Eric Jun 26 '15 at 06:53
  • Ok, I'm starting to see your point there. I'll work some more on my porting my Unit tests and come back to this question, thanks for now! – Chirlo Jun 26 '15 at 11:04
1

I have called this problem "convergent testing" but I can't figure out why or where there term comes from so take it with a grain of salt.

For any test you run the risk of the complexity of the test code approaching the complexity of the code under test.

In your case, the the code winds up being basically the same which is just writing the same code twice. Sometimes there is value in that. For example, if you are writing code to keep someone in intensive care alive, you could write it twice to be safe. I wouldn't fault you for the abundance of caution.

For other cases there comes a point where the likelihood of the test breaking invalidates the benefit of the test catching real issues. For that reason, even if it is against best practice in other ways (enumerating things that should be calculated, not writing DRY code) I try to write test code that is in some way simpler than the production code, so it is less likely to fail.

If I cannot find a way to write code simpler than the test code, that is also maintainable(read: "that I also like"), I move that test to a "higher" level(for example unit test -> functional test)

I just started playing with property based testing but from what I can tell it is hard to make it work with many unit tests. For complex units, it can work, but I find it more helpful at functional testing so far.

For functional testing you can often write the rule a function has to satisfy much more simply than you can write a function that satisfies the rule. This feels to me a lot like the P vs NP problem. Where you can write a program to VALIDATE a solution in linear time, but all known programs to FIND a solution take much longer. That seems like a wonderful case for property testing.

janicedn
  • 53
  • 5