different behavior for different "for"s in benchmark

Question

We can write a simple benchmark using google benchmark or https://www.quick-bench.com/,

static void range_based_for(benchmark::State &state) {
    for (auto _ : state) {
        std::to_string(__LINE__);
    }
}
BENCHMARK(range_based_for);

We can also rewrite it with std::for_each,

static void std_for_each(benchmark::State &state) {
    std::for_each (std::begin(state), std::end(state), [] (auto _) {
        std::to_string(__LINE__);
    });
}
BENCHMARK(std_for_each);

Everything is good. However, when we use old school for statement, it runs without finishing.

static void old_school_for(benchmark::State &state) {
    for (auto iter = std::begin(state); iter != std::end(state); ++iter) {
        std::to_string(__LINE__);
    };
}
BENCHMARK(old_school_for);

In fact, std::for_each is implemented with this style. How can they behave different?

First of all, do you build with optimization enabled? Benchmarking should always be done on optimized builds. And if you build with optimizations, then the loops would probably be no-ops, and do nothing since they have no side-effects. — Some programmer dude, May 23 '22 at 07:29
As for the problem with the third loop, look at the generated assembly code to try and figure out the problem. Use e.g. [the compiler explorer](https://godbolt.org) to help you. — Some programmer dude, May 23 '22 at 07:31
One difference between those "styles" is how many times `std::end(state)` is executed. Try caching that value in the last snippet. — Bob__, May 23 '22 at 07:56
Not only `std::for_each` is implemented that way – the range based for loop maps to something pretty similar as well, though some details differ, e.g. the range-based for loop is mandated by the standard to operate such that `begin` and `end` getting called just once (if I recall correctly) – compare Bob's comment. — Aconcagua, May 23 '22 at 07:57
One difference is that the last loop calls `std::end(state)` on every iteration, but the other two do not (a range-based `for` is specified accordingly, and the arguments of `std::for_each()` are evaluated before it is called - so `std::end(state)` is not reevaluated on each iteration). If `std::end(state)` affects members of `state` on each call [I don't know, not having read documentation or source for `benchmark::State`] that might explain the difference. — Peter, May 23 '22 at 07:59

score 5 · Accepted Answer · answered May 23 '22 at 08:52

5

The begin/end functions are documented with a warning: says "These functions should not be called directly"

These functions should not be called directly.

REQUIRES: The benchmark has not started running yet. Neither begin nor end have been called previously.

end calls StartKeepRunning

And what does StartKeepRunning do? It resets the number of iterations to the maximum

It is clear that you're only supposed to call begin and end once. The difference in your third loop is that std::end(state) is called once per iteration, which apparently resets the iteration count back to the maximum.

I have no idea why the library is designed this way.

answered May 23 '22 at 08:52

user253751

57,427
7
48
90

Thanks a lot. I precompute end instead of call end() and it runs as expected. – luckycpp May 23 '22 at 12:27
i'm the author of the library. can you explain why this is preferred to the documented first loop version? i'd like to know if i should support this use-case better. – dma May 24 '22 at 15:48
dma you need to ping @luckycpp to ask why s/he is doing this – user253751 May 24 '22 at 15:49
I was trying to replace range-based-for with tbb::parallel_for_each, and incidentally tried those other two ways. – luckycpp May 27 '22 at 09:02

different behavior for different "for"s in benchmark

1 Answers1