1

Given the following code

#include <benchmark/benchmark.h>
#include <iostream>

static void BM_foo(benchmark::State& state) {
  std::cout << "Foo "<< std::endl;
  for (auto _: state) {
    std::cout <<  state.iterations()  << " In loop " <<std::endl;
  }
}

BENCHMARK(BM_foo);

BENCHMARK_MAIN();

I thought the std::cout << "Foo "<< std::endl; will be executed only once, but during my test, it runs 7 times.

So my questions are:

  1. How can I run some setup code only once before the benchmark, like "std::cout << "Foo "<< std::endl"?
  2. when I use "state.iterations()", I want to get "1, 2, 3 .." one by one, but i aways get 0. how can I get the seq number of the iterators?

In my real case, I want to run some storage engine related code in multi threads scenario. the steps like this:

  1. open connection to the storage engine. let's assume the connection object CAN be shared among different threads.
  2. open session based on connection. let's assume the session object CAN NOT be shared among threads.
  3. run many db_related ops, which is the only code I want to benchmark.
  4. close session, connection.

I have the following pseudo code, but I thought it 2 issues at least.

DBConnection* conn; // global variables, so that each thread can run access it.

static void BM_db_insert(benchmark::State& state) {
  if (state.thread_index == 0) {
    # only let 1 thread to setup the DBConnections
    conn = openDBConn();
  }
  DBSession* session = conn.openSession();
  for (auto _ : state) {
    session.insertOp(id, ..); 
  }
  session.close();
  if (state.thread_index == 0) {
    conn.close();
  }
}

Issues:

  1. Shall I have to use global variables DBConnection here so that it can be accessed by each thread?
  2. Suppose "DBSession* session = conn.openSession();" can be executed before the "conn is really set up", and I also don't want to benchmark openSession every time, how shall I fix this issue?

So totally I have 4 questions in 2 parts, and if you have more suggestion of my code, that will be much better. Thanks!

zhihuifan
  • 1,093
  • 2
  • 16
  • 30
  • It looks a lot like what you need is a profiling library, rather than a miro-benchamraking one. –  May 19 '18 at 13:59

1 Answers1

1

  1. We run the entire benchmark multiple times to find the optimum number of iterations to run, however that preamble is being run only once per benchmark run. You're doing that right.

  2. You need to track the iteration count yourself if you need that. However, if you do, then maybe what you're benchmarking isn't agnostic to the number of iterations, which is determined dynamically. If you want a fixed number of iterations you should look at State::KeepRunningBatch.

  1. Having the connection global works, or static member of a fixture.

  2. I'd add an explicit wait for the connection to be set up on every thread. Some sort of conn.IsReady() in a loop before you open the session.

dma
  • 1,758
  • 10
  • 25
  • Thanks, but for Q1: the code "std::cout << "Foo "<< std::endl" runs more than once acturally. that looks strange to me. I build the code with the latest benchmark from github directly, commit ID: e90801ae475f23877319d67b96c07bf0c52405f3 – zhihuifan May 25 '18 at 10:18
  • We call the entire benchmark function multiple times with different iteration counts governing the inner loop. If your inner loop was more expensive you'd see Foo printed fewer times, as we'd reach the minimum runtime sooner. It's working as intended. – dma Aug 02 '18 at 21:23
  • 2
    @dma: What if the preamble is also expensive and I don't want to spend time re-running it? How can I make sure that the preamble runs only once? – Unni Oct 28 '19 at 06:20