Haskell unit testing

Question

I'm new to haskell and working on unit testing, however I find the ecosystem to be very confusing. I'm confused as to the relationship between HTF and HUnit.

In some examples I see you set up test cases, export them in an tests list, and then run in ghci with runTestsTT (like this HUnit example).

In other examples, you create a test runner tied into the cabal file that uses some preprocessor magic to find your tests like in this git example. Also it seems that HTF tests need to be prefixed with test_ or they aren't run? I had a hard time finding any documentation on that, I just noticed the pattern that everyone had.

Anyways, can someone help sort this out for me? What is considered the standard way of doing things in Haskell? What are the best practices? What is the easiest to set up and maintain?

Answer 1

Generally, any significant Haskell project is run with Cabal. This takes care of building, distribution, documentation (with the help of haddock), and testing.

The standard approach is to put your tests in the test directory and then set up a test suite in your .cabal file. This is detailed in the user manual. Here's what the test suite for one of my projects looks like

Test-Suite test-melody
  type:               exitcode-stdio-1.0
  main-is:            Main.hs
  hs-source-dirs:     test
  build-depends:      base >=4.6 && <4.7,
                      test-framework,
                      test-framework-hunit,
                      HUnit,
                      containers == 0.5.*

Then in the file test/Main.hs

import Test.HUnit
import Test.Framework
import Test.Framework.Providers.HUnit
import Data.Monoid
import Control.Monad
import Utils

pushTest :: Assertion
pushTest = [NumLit 1] ^? push (NumLit 1)

pushPopTest :: Assertion
pushPopTest = [] ^? (push (NumLit 0) >> void pop)

main :: IO ()
main = defaultMainWithOpts
       [testCase "push" pushTest
       ,testCase "push-pop" pushPopTest]
       mempty

Where Utils defines some nicer interfaces over HUnit.

For lighter-weight testing, I strongly recommend you use QuickCheck. It lets you write short properties and test them over a series of random inputs. For example:

 -- Tests.hs
 import Test.QuickCheck

 prop_reverseReverse :: [Int] -> Bool
 prop_reverseReverse xs = reverse (reverse xs) == xs

And then

 $ ghci Tests.hs
 > import Test.QuickCheck
 > quickCheck prop_reverseReverse
 .... Passed Tests (100/100)

Answer 2

I'm also newbie haskeller and I've found this introduction really helpful: "Getting started with HUnit". To summarize, I'll put here simple testing example of HUnit usage without .cabal project file:

Let's assume that we have module SafePrelude.hs:

module SafePrelude where

safeHead :: [a] -> Maybe a
safeHead []    = Nothing
safeHead (x:_) = Just x

we can put tests into TestSafePrelude.hs as follows:

module TestSafePrelude where

import Test.HUnit
import SafePrelude

testSafeHeadForEmptyList :: Test
testSafeHeadForEmptyList = 
    TestCase $ assertEqual "Should return Nothing for empty list"
                           Nothing (safeHead ([]::[Int]))

testSafeHeadForNonEmptyList :: Test
testSafeHeadForNonEmptyList =
    TestCase $ assertEqual "Should return (Just head) for non empty list" (Just 1)
               (safeHead ([1]::[Int]))

main :: IO Counts
main = runTestTT $ TestList [testSafeHeadForEmptyList, testSafeHeadForNonEmptyList]

Now it's easy to run tests using ghc:

runghc TestSafePrelude.hs

or hugs - in this case TestSafePrelude.hs has to be renamed to Main.hs (as far as I'm familiar with hugs) (don't forget to change module header too):

runhugs Main.hs

or any other haskell compiler ;-)

Of course there is more then that in HUnit, so I really recommend to read suggested tutorial and library User's Guide.

Answer 3

You've had answers to most of your questions, but you also asked about HTF, and how that works.

HTF is a framework that is designed for both unit testing -- it is backwards compatible with HUnit (it integrates and wraps it to provide extra functions) -- and property-based testing -- it integrates with quickcheck. It uses a preprocessor to locate tests so that you don't have to manually build a list. The preprocessor is added to your test source files using a pragma:

{-# OPTIONS_GHC -F -pgmF htfpp #-}

(alternatively, I guess you could add the same options to your ghc-options property in your cabal file, but I've never tried this so don't know if it is useful or not).

The preprocessor scans your module for top-level functions named test_xxxx or prop_xxxx and adds them to a list of tests for the module. You can either use this list directly by putting a main function in the module and running them (main = htfMain htf_thisModuleTests) or export them from the module, and have a main test program for multiple modules, which imports the modules with tests and runs all of them:

import {-@ HTF_TESTS @-} ModuleA
import {-@ HTF_TESTS @-} ModuleB
main :: IO ()
main = htfMain htf_importedTests

This program can be integrated with cabal using the technique described by @jozefg, or loaded into ghci and run interactively (although not on Windows - see https://github.com/skogsbaer/HTF/issues/60 for details).

Tasty is another alternative that provides a way of integrating different kinds of tests. It doesn't have a preprocessor like HTF, but has a module that performs similar functions using Template Haskell. Like HTF, it also relies on naming convention to identify your tests (in this case, case_xxxx rather than test_xxxx). In addition to HUnit and QuickCheck tests, it also has modules for handling a number of other test types.