Caliper: Interpreting the benchmark results

Question

Can anyone help me with this question. I write simple benchmark with single method

public void timeIntArrayBlockingQueue(int reps) {
  for (int i = 0; i < reps; i++) {
    for (int j = 0; j < length; j++)
       queue.add(data[j]);
       Integer value;
       while ((value = queue.poll()) != 101) {}
    }
}

for ArrayBlockingQueue and get result:

0% Scenario{vm=java, trial=0, benchmark=IntArrayBlockingQueue, length=100} 5480.41 ns; σ=311.51 ns @ 10 trials

33% Scenario{vm=java, trial=0, benchmark=IntArrayBlockingQueue, length=1000} 56579.97 ns; σ=5530.23 ns @ 10 trials

67% Scenario{vm=java, trial=0, benchmark=IntArrayBlockingQueue, length=10000} 553782.25 ns; σ=27212.56 ns @ 10 trials

length     us linear runtime
   100   5.48 =
  1000  56.58 ===
 10000 553.78 ==============================

vm: java
trial: 0
benchmark: IntArrayBlockingQueue

What this digits (5480.41ns 56579.97ns 553782.25ns) mean exactly.

0% Scenario method timeIntArrayBlockingQueue was called 10 times with param length = 100.
(timeIntArrayBlockingQueue(100) - 10 time )

33% Scenario method timeIntArrayBlockingQueue was called 10 times with param length = 1000. (timeIntArrayBlockingQueue(1000) - 10 time ) . .

How to interpreting the results of this benchmark ?

Answer 1

The percentage is just the progress, i.e., nothing interesting afterwards.

What this digits (5480.41ns 56579.97ns 553782.25ns) mean exactly.

This is the average duration of single iteration, i.e., the duration of timeXXX divided by rep.

The σ=311.51 ns is the Standard deviation. Sort of absolute error, if σ/measured_time gets close to one, you can be sure that's something's wrong.

benchmark=IntArrayBlockingQueue, length=100

That's the name of the benchmark are the parameters given, e.g. via @Param.

---

The important part

Do not let you timeXXX method return nothing, instead compute and return some result so that you can be sure that not the whole computation got optimized away. The body of you method should be something like

...
while ((value = queue.poll()) != 101) {
    result += value;
}

The single addition together with Integer.getValue() can hardly make it much slower. It does a bit, but the benchmark get a bit more realistic this way (you use the polled value, don't you). Normally you want to run multiple benchmarks and compare their result so you can choose the fastest variant (Actually, normally you don't need any benchmark).

You loop is strange, anyway, there is normally just one element in the queue...