Initialize benchmark in criterion and exclude initialization time from results

Question

I need to benchmark some code inside IO, and criterion supports that pretty well. But I want to perform few initialization steps (different for each benchmark). The naive approach:

main = defaultMain
  [ bench "the first" $ do
      initTheFirst
      theFirst
      cleanUpTheFirst
  , bench "the second" $ do
      initTheSecond
      theSecond
      cleanUpTheSecond
  ]

But it performs initialization and cleanup for every benchmark run (100 times by default) and includes initialization time to final results. Is it possible to exclude initialization time?

ADDED: The code uses global state (mongodb actually), so I can't prepare two initial states simultaneously.

score 5 · Answer 1 · answered Oct 02 '13 at 19:22

Here is a solution using custom main as suggested by argiopeweb:

import Control.Monad
import Control.Monad.IO.Class
import Control.Concurrent
import Criterion
import Criterion.Config
import Criterion.Monad
import Criterion.Environment

main :: IO ()
main = myMain [
  (initTheFirst, theFirst),
  (initTheSecond, theSecond)
  ]

initTheFirst :: IO ()
initTheFirst = do
  putStrLn "initializing the first"
  threadDelay 1000000

theFirst :: Benchmark
theFirst = bench "the first" $ do
  return () :: IO ()

initTheSecond :: IO ()
initTheSecond = do
  putStrLn "initializing the second"
  threadDelay 1000000

theSecond :: Benchmark
theSecond = bench "the second" $ do
  return () :: IO ()

myMain :: [(IO (), Benchmark)] -> IO ()
myMain benchmarks = withConfig defaultConfig $ do
  env <- measureEnvironment
  forM_ benchmarks $ \(initialize, benchmark) -> do
    liftIO $ initialize
    runAndAnalyse (const True) env benchmark

The output:

warming up
estimating clock resolution...
mean is 1.723574 us (320001 iterations)
found 1888 outliers among 319999 samples (0.6%)
  1321 (0.4%) high severe
estimating cost of a clock call...
mean is 43.45580 ns (13 iterations)
found 2 outliers among 13 samples (15.4%)
  2 (15.4%) high severe
initializing the first

benchmarking the first
mean: 7.782388 ns, lb 7.776217 ns, ub 7.790563 ns, ci 0.950
std dev: 36.01493 ps, lb 29.29834 ps, ub 52.51021 ps, ci 0.950
initializing the second

benchmarking the second
mean: 7.778543 ns, lb 7.773192 ns, ub 7.784518 ns, ci 0.950
std dev: 28.85100 ps, lb 25.59891 ps, ub 32.85481 ps, ci 0.950

You can see that init* functions are called only once and don't affect benchmarks results.

Elliot Robinson · Accepted Answer · 2013-10-02T17:35:59.020

2

Three real options I can see here. Either initialize and cleanup before and after:

main = do
  initTheFirst
  initTheSecond

  defaultMain
    [ bench "the first" theFirst
    , bench "the second" theSecond
    ]

  cleanUpTheFirst
  cleanUpTheSecond

Or, if that isn't possible, also benchmark your cleanup and initialization processes and modify your benchmark times accordingly.

Alternatively, you could forego using the provided defaultMain and instead roll your own using the lower-level functions provided by Criterion.

edited Oct 02 '13 at 17:35

answered Oct 02 '13 at 15:54

Elliot Robinson

1,384
7
16

AFAIU, the requirement is to cleanup in between the two benchmarks. – Nikita Volkov Oct 02 '13 at 16:04
Hence my second option. If there is legitimately enough global state here that `initTheFirst` is necessary before every `theFirst >> cleanUpTheFirst`, I don't see an alternative to one of these two options while still using defaultMain. – Elliot Robinson Oct 02 '13 at 16:11
I can't use the first option unfortunately. Re the second option: do you mean just substitute run time of init/cleanup pair from total run time? Will not it invalidate results because of variations in init/cleanup run time (it is heavy)? – Yuras Oct 02 '13 at 16:16
Custom main instead of the default one seems to be the way to go. Thanks for idea! – Yuras Oct 02 '13 at 16:18
@argiopeweb Could you please add a note about custom main as the 3d option into your answer? (I don't know how I missed the possibility to create custom main... but it is exactly what I need, works like a charm) – Yuras Oct 02 '13 at 16:36
Done. I'd love to see your results. Any chance you could make it available here or as a GitHub Gist? – Elliot Robinson Oct 02 '13 at 17:37

Initialize benchmark in criterion and exclude initialization time from results

2 Answers2