0

I have a few benchmarks that would like to test using criterion. The problem is that the function I am trying to benchmark may take a very long time(100 seconds).

My setup is something like this

benchmarks128StatesMediumFormulas =
[
    bench "Testing for transition system t128StatesAgents1 and feasy"
          (nf (myFuntion x1 x2) 2),
    bench "Testing for transition system t128StatesAgents2 and feasy"
          (nf (myFunction x1 x2 feasy) 2)
]

where my function is the function I need to benchmark and x1 and x2 are defined somewhere else in the file.

Then I do

main :: IO ()
main = defaultMain
  [
    bgroup "for one agent easy:" benchmarks128StatesEasyFormulas
]

Then i get an output of something like

benchmarking 128 states and easy formulas:/Testing for transition system t128StatesAgents2 and fEasy1
time                 115.5 s    (111.1 s .. 119.2 s)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 115.7 s    (115.0 s .. 116.7 s)
std dev              983.9 ms   (389.9 ms .. 1.263 s)
variance introduced by outliers: 19% (moderately inflated)

The problem is that this more then an hour to produce that outcome.

Is this because criterion is running too many sample?? Or is there some other problem?? Can I work around this??

I read something about --time-limit option but did not quite understand how it works. If the solution revolves around using --time-limit how can I pass this argument when using stack benck?

arrowd
  • 33,231
  • 8
  • 79
  • 110
Augusto Peres
  • 61
  • 5
  • 2
    One hour only gets you about 35 copies of 100 seconds. Demanding the ability to run your thing a couple dozen times seems completely reasonable to me. If you're not willing to wait that long, can I suggest that you benchmark smaller pieces of your function independently? Catching performance regressions in the components that make up `myFunction` is almost certainly enough to catch performance regressions in `myFunction`, and more informative about where to start debugging them to boot. – Daniel Wagner Apr 25 '20 at 23:28
  • Can't tell what your `myFunction x1 x2` does but in case it takes different length of time to finalize depending on the complexity of the provided data then just provide simple input to establish a minimal test of that function. The size of the data shouldn't effect the time and space complexity of the function. In this case Criterion can take necessary number of samples in a more resonable time. – Redu Apr 26 '20 at 05:26
  • @Redu is there some way I can sacrifice accuracy for speed in larger inputs on my function? – Augusto Peres Apr 26 '20 at 10:30
  • You may try `stack bench --ba --time-limit=0.1`. Note that `--time-limit` option should have a value in the range 0.1 ~ 86400.0. – Redu Apr 26 '20 at 11:28

0 Answers0