How to benchmark custom interface using google benchmark

Question

since I can't find really good resource or documentation besides what's on the github readme, I'll ask the community here.

Assuming that I have an interface called Base, and multiple derived types.
I want to benchmark a virtual function Foo of all derived types with the same function and the same argument passed to Foo.
For each derived type I want to call Foo multiple times in a single benchmark with varying arguments passed from an std::vector<std::size_t>.

Also some of my constructor of derived types takes extra arguments, how should I deal with it?

The easiest thing to do as I see it, is to use templated benchmark and pass to:

const std::vector<std::size_t> v {1, 2, 4, 8, 16};
BENCHMARK_TEMPLATE(func, Derived1, v);
BENCHMARK_TEMPLATE(func, Derived2, v);
BENCHMARK_TEMPLATE(func, Derived3, v);
...

I don't feel this is the correct way to do this since for each derived type with different constructor signature I would need to have different benchmark function which is almost identical to the previous one.

What would be the proper way to benchmark this?

**UPDATE*: The way I solved this is as follows:

std::unique_ptr<Base> GetObj(const std::size_t index)
{
    switch(index)
    {
    case 0:
        return std::make_unique<Derived1>(...);
    case 1:
        return std::make_unique<Derived2>(...);
    ...
    }
}

void func(benchmark::State& state)
{
    auto r = state.range();
    auto object = GetObj(r);
    for (auto _ : state)
    {
        ...
    }
}
BENCHMARK(func)->DenseRange(0,5);

But what I don't like about it is that the name of the benchmark tests is of the pattern func/0-5 and I would like it to be func/Derived1, func/Derived2 and so on.. any ideas how to achieve that?

Solutions are answers, not questions. Please post your solution as an answer, and remove it from the question. **YES**, you are encouraged to answer your own questions! — Kuba hasn't forgotten Monica, Jun 02 '20 at 18:57
@ReinstateMonica Unless I can overcome this little annoyance caused by my solution, I do not wish to make it the best answer or propose it. I post it as an update to my OP so others could see my progress and build upon it. If there's another solution that doesn't introduce this issue then I will be accepting it as the solution, otherwise I'm just updating my question since in my eyes it's not complete. — Jorayen, Jun 02 '20 at 19:04

score 1 · Accepted Answer · answered Jun 01 '20 at 14:44

Your approach is pretty much correct. If you want to avoid the multiple benchmark functions, move the core of the benchmark to a separate function and call it from each of the per-type ones. Something like:

static void Benchmark(Base* b, benchmark::State& st) {
  for (auto _ : state) {
    b->DoTheThing()
  }
}

void BM_Foo(benchmark::State& st) {
  Foo f;
  Benchmark(&f, st);
}

BENCHMARK(BM_Foo);

If you want to be more sophisticated, you could use the RegisterBenchmark function to register things more flexibly. Something like:

auto BM_base = [](benchmark::State& st, auto inputs, Base* b) { /* ... */ };

int main(int argc, char**argv) {
  const std::vector<std::size_t> v = {1, 2, 4, 8, 16};
  benchmark::RegisterBenchmark("foo", BM_base, v, new Foo());
  benchmark::Initialize(&argc, argv);
  return benchmark::RunSpecifiedBenchmarks();
}

As an aside, should v be set as the Args when the benchmark is registered?

I ended up having one benchmark function that pulls pointers to base from a helper function based on index. The index is passed using `DenseRange`, this way I can have only one function. — Jorayen, Jun 01 '20 at 23:06
Hey, I've updated my question with my solution and also a new question that arose from this implementation. Would love to hear your opinion on how to achieve what I'm looking for on my updated question. — Jorayen, Jun 02 '20 at 18:25
you should be using templated benchmarks or one of the solutions in my answer which will give you the naming you want. — dma, Jun 03 '20 at 07:56

How to benchmark custom interface using google benchmark

1 Answers1