1

I have interest in using the quick check library but it seems that it is designed to test properties. What I would like to do is generate random data for my defined data types and test functions I have written. I do not care about what the result is, just if the function produces a runtime error when fed random data. All of the quick check examples I have seen are for testing properties of functions like is the result greater than 5 when fed random data. Is there a way to use quick check in this manner? Something like

data Result = A | B

fun :: Result -> Int
fun A = 5

main = check fun

In the above code I have a custom data type, and a incomplete function. This function will fail if passed B. There are of course more types of runtime errors than just incomplete functions. I would like to have check generate data, and feed it to the function. Not caring what the result is. Is quick check able to do this?

Edit - I should note that I am not looking for flags that check for incomplete patterns, and what not. I am interested in general purpose runtime error checks.

44701
  • 397
  • 4
  • 10
  • 2
    This error should get cought at compile time, e.g. `-fwarn-incomplete-patterns -Werror` (https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/options-sanity.html). Depending on what your other errors are, `-Wall -Werror` could be sufficient, even before a single QuickCheck test runs. – Zeta Feb 23 '16 at 22:48
  • Thanks but I am aware of such flags. I am interested in general purpose runtime error checking. – 44701 Feb 23 '16 at 22:50
  • 2
    See [Test.QuickCheck.Monadic](https://hackage.haskell.org/package/QuickCheck-2.8.2/docs/Test-QuickCheck-Monadic.html), which allows for the testing of arbitrary `IO` wrapped values. You can check for runtime errors using `Control.Exception.catch` or the like. In this specific case, you would want to catch the `PatternMatchFail` exception - but of course you can check for any exception you like. – user2407038 Feb 23 '16 at 23:01
  • What I mean is to ensure that there are no errors. I am not interested in specific errors, or looking for custom errors I define. If that is what you are trying to say I apologies for misunderstanding. – 44701 Feb 23 '16 at 23:16
  • 2
    I haven't thought it through, but could you use a dummy prop like `dummy f = \x -> seq (f x) True` in combination with `catch`? – epsilonhalbe Feb 24 '16 at 00:08
  • @epsilonhalbe, the dummy you seek is `Control.Exception.evaluate` – dfeuer Feb 24 '16 at 00:51
  • 3
    This is not the Haskell way. Convert your partial functions into total ones with a codomain that has an observable failure; then it is trivial to write a quickcheck property that failure is not observed. I make this a comment rather than an answer as trying to influence the reader to change his question in such a fundamental regard is not the StackOverflow way. ;-) – Daniel Wagner Feb 24 '16 at 01:30
  • There are all kinds of functions, and situations in Haskell where testing like this is nice. Real world Haskell is not always perfect. Just look at the standard library there are all kinds of partial functions, and functions that can fail at run time. Index bounds errors are another common one. When larger programs are written functions may expect data to be encoded in a way that cannot easily be represented in the type. This leads to partial functions or strangely typed functions where maybe is inconvenient. – 44701 Feb 24 '16 at 02:56
  • @HarpoRoeder, lots of real world Haskell *bugs* are caused by programmers trying to take the easy way out using partial functions, calling `error` to report user errors, etc. They think these are all fine for days, weeks, or months till they run into some weird error they can't seem to debug, some wacky space leak, etc., and then someone has to dig through the whole pile of crud and redo it properly. Obviously, there's a balance, and some code will need to have (hopefully-unreachable) errors, but that should only be done with due consideration. – dfeuer Feb 24 '16 at 16:01

1 Answers1

2

Rather make sure that functions that don't handle IO don't throw exceptions if possible. Exceptions can only get caught in IO, and they kind of break the things you otherwise expect from a pure function.

It's possible with some helpers from Control.Exception and Test.QuickCheck.Monadic:

> import           Control.Exception       (Exception, PatternMatchFail,
>                                           SomeException, evaluate,
>                                           fromException, try)
> import           Data.Either             (either)
> import           Test.Hspec              (anyException, describe, hspec, it,
>                                           shouldThrow)
> import           Test.QuickCheck         hiding (Result)
> import           Test.QuickCheck.Monadic (assert, monadicIO, run)

For starters, let's write a function that enables us to check that a certain exception is thrown:

> throwsExceptionOr :: Exception e => (e -> Bool) -> (a -> Bool) -> a -> IO Bool
> throwsExceptionOr pe pa = fmap (either pe pa) . try . evaluate

This enables you to write tests like this:

> prop_fun_1 x = monadicIO . run $
>   throwsExceptionOr (const True :: SomeException -> Bool)
>                     (== 5)
>                     (foo x)

throwsExceptionOr is very general, so that you can define your own helpers:

> -- | Should always throw an exception.
> throwsException :: Exception e => (e -> Bool) -> a -> IO Bool
> throwsException p = fmap (either p (const False)) . try . evaluate

> -- | Should either pass the test or throw an exception.
> exceptionOr :: a -> (a -> Bool) -> IO Bool
> exceptionOr x p = fmap (either anyException p) . try . evaluate $ x
>   where
>     anyException :: SomeException -> Bool
>     anyException = const True

Now you can write your tests as usual:

> data Result = A | B deriving (Enum, Show, Eq, Ord, Bounded, Read)
> 
> instance Arbitrary Result where
>   arbitrary = elements [A, B]
> 
> foo :: Result -> Int
> foo A = 5
> 
> prop_foo x = monadicIO . run $ foo x `exceptionOr` (== 5)

You can go further and move the monadicIO . run into another helper, but that's left as an exercise. Furthermore, you can make the functions compatible with other testing frameworks, such as hspec or tasty:

> main :: IO ()
> main = hspec $ do
>   describe "foo" $ do
>     it "either returns five or throws a pattern match fail" $ propertyIO $ \x ->
>       throwsExceptionOr patternMatchFail (==5) (foo x)
> 
>     it "throws an exception on input B" $
>       evaluate (foo B) `shouldThrow` anyException
> 
>  where
>   patternMatchFail :: PatternMatchFail -> Bool
>   patternMatchFail _ = True
> 
>   -- I think there is a better combinator for this
>   propertyIO f = property $ \x -> monadicIO . run $ f x

That being said, regardless of the used languages you want to get rid of possible runtime errors at compile time if possible. This includes getting rid of partial functions or possible type nonsense. This depends on the actual use, of course. If you can verify that head always gets called on a non-empty list throughout your program, go ahead and use it. If you can't, use pattern matching (see this discussion on head's type).

Either way, given that older versions of GHC don't provide stack calls, you rather want to have errors at compile time than errors without a stack trace at runtime (recent versions of GHC have some nice features for this).

Zeta
  • 103,620
  • 13
  • 194
  • 236