When does it make sense to use Statistics#compareTo(Statistics, double) to compare Statistics?

Question

I would like to know in which circumstances it is interesting to compare Statistics using a confidence thanks to the method Statistics#compareTo(Statistics, double). Does it make sense to use this method to compare the performances of 2 approaches? What is the exact purpose/meaning of confidence in this method?

Answer 1

As javadoc states, it's value of confidence interval.

Roughly speaking, assume you have two distributions (aka benchmark runs) of two methods run times. By default, JMH assumes it's normal distribution with some parameters (mean and variance). But distributions aren't numbers: you can't compare one mean with another and say "Hey, first mean is smaller than second one, thus first approach is better in average!". First you should prove that they don't belong to the same distribution (it's still possible even if they (samples, not distributions) have slightly different means), otherwise such comparison makes no sense. For such proves special statistical test is used. But as long as test is dealing with samples, not actual distributions, test can't say "Two data sets don't belong to the same distribution", test can only say "Two data sets don't belong to the same distribution with 99% probability". This 99% (or any other) actually is confidence you are asking.

So, basically, s1.compareTo(s2, 0.9) == 1 means that s1 (first benchmark run times) has smaller average run time than second one with 90% probability (and with 10% there could be any other result, not only the opposite).