Testing not equal in Quickcheck?

Question

I'm new to QuickCheck and can't quite wrap my head around how to use it.

Let's say I accidentally implemented a data-type with a Set (instead of a List):

data Profile = Profile (Set Strategy)
--for completeness:
data Strategy = Strategy Int

and then ran into this bug later, where two objects are equal even if they shouldn't:

Profile (Set.fromList [1,2,3]) == Profile (Set.fromList [2,1,3])
-- D'OH! Order doesn't matter in sets!

How can I write a QuickCheck test case to test for this case? In pseudo-code this would look something like this:

assertNotEqual(Profile (Set.fromList [1,2,3]), Profile (Set.fromList [2,1,3]))
assertEqual(Profile (Set.empty), Profile (Set.empty ))

I've tried looking at the examples on the project's github, but it seems they don't cover such trivial cases.

Any hints welcome!

The problem I have with answering this question sensably via QC is you didn't give a full ADT. You see the definition of `Profile` instead of an opaque type, builder function, and transformations. If you were given a proper ADT then you could simply decide on which properties should hold after each operation and test for those. — Thomas M. DuBuisson, Feb 17 '13 at 18:16

score 3 · Answer 1 · answered Feb 17 '13 at 08:26

You can do it using existential quantification, supported by SmallCheck:

> depthCheck 5 $ exists $ \xs -> (xs :: [Integer]) /= sort xs
    Depth 5:
      Completed 1 test(s) without failure.
> depthCheck 5 $ exists $ \xs -> fromList (xs :: [Integer]) /= fromList (sort xs)
    Depth 5:
      Failed test no. 1. Test values follow.
      non-existence

Another option, using universal quantification (also works in QuickCheck):

> smallCheck 5 $ \xs ys -> xs /= ys ==> fromList (xs :: [Integer]) /= fromList ys
Depth 0:
  Completed 1 test(s) without failure.
  But 1 did not meet ==> condition.
Depth 1:
  Completed 4 test(s) without failure.
  But 2 did not meet ==> condition.
Depth 2:
  Failed test no. 26. Test values follow.
  [0]
  [0,0]

score 3 · Answer 2 · answered Feb 17 '13 at 14:54

How can I write a QuickCheck test case to test for this case?

You shouldn't! QuickCheck is a tool for property based testing. In property based testing, you give a property of your data structure (or whatever) and the testing tool will automatically generate test cases to see if that property holds for the generated test cases. So, let's see how you can give a property instead of giving concrete test cases like [1,2,3] and why properties are advantageous!

So. I started off with

import Test.QuickCheck
import qualified Data.Set as Set 
import Data.Set (Set)

data Profile = Profile (Set Int)
  deriving (Eq, Show)

mkProfile :: [Int] -> Profile
mkProfile = Profile . Set.fromList

-- | We will test if the order of the arguments matter.
test_mkProfile :: [Int] -> Bool
test_mkProfile xs = (mkProfile xs `comp` mkProfile (reverse xs))
  where comp | length xs <= 1 = (==)
             | otherwise      = (/=)

This is how I reasoned for my property: Well, for empty and singleton list case, then reverse is just the identity, so we expect mkProfile xs be the same as mkProfile (reverse xs). Right? I mean mkProfile gets exactly the same argument. In the case that length xs >= 2 then reverse xs clearly isn't xs. Like reverse [1, 2] /= [2, 1]. And we know that a Profile do care about the order.

Now lets try this out in ghci

*Main> quickCheck test_mkProfile 
*** Failed! Falsifiable (after 3 tests and 1 shrink):     
[0,0]

Note now that there actually are two mistakes in our code. One, First, Profile should be using a list and not a set. Second, our property is wrong! Because even if length xs >= 2, xs == reverse (xs) can be true. Let's try to fix the first error and see how quickcheck will still point out the second flaw.

data Profile2 = Profile2 [Int]
  deriving (Eq, Show)

mkProfile2 :: [Int] -> Profile2
mkProfile2 = Profile2 

-- | We will test if the order of the arguments matter.
test_mkProfile2 :: [Int] -> Bool
test_mkProfile2 xs = (mkProfile2 xs `comp` mkProfile2 (reverse xs))
  where comp | length xs <= 1 = (==)
             | otherwise      = (/=)

Remember, our code is correct now but our property flawed!

*Main> quickCheck test_mkProfile2
+++ OK, passed 100 tests.
*Main> quickCheck test_mkProfile2
+++ OK, passed 100 tests.
*Main> quickCheck test_mkProfile2
+++ OK, passed 100 tests.
*Main> quickCheck test_mkProfile2
+++ OK, passed 100 tests.
*Main> quickCheck test_mkProfile2
+++ OK, passed 100 tests.
*Main> quickCheck test_mkProfile2
+++ OK, passed 100 tests.
*Main> quickCheck test_mkProfile2
+++ OK, passed 100 tests.
*Main> quickCheck test_mkProfile2
*** Failed! Falsifiable (after 8 tests):                   
[-8,-8]

Yes. You still need to think! Or you might get the false impression that everything is ok just because your code literally passed 700 test cases! Ok now lets fix our property too!

test_mkProfile2_again :: [Int] -> Bool
test_mkProfile2_again xs = (mkProfile2 xs `comp` mkProfile2 ys)
  where ys   = reverse xs
        comp | xs == ys  = (==)
             | otherwise = (/=)

Now lets see that it works multiple times!

*Main> import Control.Monad
*Main Control.Monad> forever $ quickCheck test_mkProfile2_again
+++ OK, passed 100 tests.
+++ OK, passed 100 tests.
+++ OK, passed 100 tests.
+++ OK, passed 100 tests.
+++ OK, passed 100 tests.
+++ OK, passed 100 tests.
+++ OK, passed 100 tests.
... (a lot of times)

Hooray. We've now not only squashed the bug in our Profile implementation, but also got a much better understanding of our code and the properties it adheres too!

But wouldn't this mean that we are testing that `[1,1] \= [1,1]`? Which should still be true... (or does quickcheck always generate lists with distinct elements?) — Andriy Drozdyuk, Feb 17 '13 at 17:41
@drozzy Well the focus on my answer wasn't really on your application. I interpreted the correct behavior of `==` on a `Profile` freely. But yes, quickcheck will generate all kinds of lists, even those with non-distinct elements (which should've been clear from the failing cases in the answer). — Tarrasch, Feb 17 '13 at 22:08
@drozzy I think the point here is that Tarrasch's change to the specification means that palindromes like [1, 1] generate the equality test (==) rather than inequality (/=); and this is exactly what is needed. I'm a newcomer to QuickCheck myself. — fairflow, Feb 18 '13 at 13:45

score 2 · Accepted Answer · answered Feb 17 '13 at 18:46

2

As I commented, my main issue answering this question is the lack of structure around your Profile type. If you define Profile, a set of operations, and invariants then it becomes easy to make quickcheck tests.

For example, lets say you have a Profile, a way to build profiles, and one way to modify profiles. The properties will all be uniqueness

 module Profile (Profile, mkProfile, addItem) where
 import Data.Set

 newtype Profile = Profile { unProfile :: Set Int }
   deriving (Eq, Ord, Show)

 mkProfile :: [Int] -> Profile
 mkProfile = Profile . fromList

 addItem :: Int -> Profile -> Profile
 addItem x = Profile . insert x . unProfile

you could test such an ADT with quickcheck by stating properties before and after each operation:

 import Test.QuickCheck
 import Profile as P

 prop_unique_list_unique_profile :: [Int] -> [Int] -> Bool
 prop_unique_list_unique_profile xs ys =
    xs /= ys ==> mkProfile xs /= mkProfile ys

 prop_addItem_nonequal :: Int -> [Int] -> Bool
 prop_addItem_nonequal x xs = P.addItem x xs /= xs

answered Feb 17 '13 at 18:46

Thomas M. DuBuisson

64,245
7
109
166

Thanks a lot! Even though I didn't use the `mkProfile` and `addItem` functions, I managed to write the test for what I wanted, by using the `==>` operation which I didn't know about before! P.S.: Shouldn't your prop functions return `Property` instead of `Bool`? – Andriy Drozdyuk Feb 17 '13 at 21:52
Also, I rewrote it with guards as `prop_blah xs ys` with `| xs /= ys = Profile xs \= Profile ys` and `| xs == ys = Profile xs == Profile ys` as guards. Thanks – Andriy Drozdyuk Feb 17 '13 at 21:56
drozzy: Yes, `Property` would be right. By the `wrong` tests passing do you mean tests that are false are passing? You can try to increase the number of test cases ran. I know a number of projects set the number of tests at 30000 (up from the default of 100). – Thomas M. DuBuisson Feb 17 '13 at 23:18
Yes, that was the problem! It didn't generate any "false" cases. Anyways - after increasing the number it gave me some failures. Also, I decided to make this into an HUnit test, and combine it with quickcheck with [test-framework](https://github.com/batterseapower/test-framework/blob/master/example/Test/Framework/Example.lhs). – Andriy Drozdyuk Feb 18 '13 at 00:33

Testing not equal in Quickcheck?

3 Answers3