BenchmarkDotNet running same benchmark multiple times

Question

Faced with the issue that my current configuration for the benchmarks did not working properly, the reason of it is that I'm trying to run the benchmark only ones, for that as it mentioned in documentation I've to use this attribute [SimpleJob(RunStrategy.ColdStart, targetCount: 1)] and this was getting me in the wrong direction because from the console I've notice that my single bench was founded twice.

// ***** BenchmarkRunner: Start   *****
// ***** Found 2 benchmark(s) in total *****
// ***** Building 2 exe(s) in Parallel: Start   *****

internal class Program
{
    static async Task Main(string[] args) => BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args, new DebugInProcessConfig());
}

[MarkdownExporter]
[AsciiDocExporter]
[HtmlExporter]
[CsvExporter]
[RPlotExporter]
[SimpleJob(RunStrategy.ColdStart, targetCount: 1)]
public class MyBench
{
    [Params(2)] public int _anotherValueToTestWith;

    [Params(2)] public int _valueToTestWith;

    [GlobalSetup]
    public void GlobalSetup()
    {
        // ...
    }

    [GlobalCleanup]
    public void GlobalCleanup()
    {
        // CleanUp
    }

    [Benchmark]
    public void AccessTokenServiceBench()
    {
        // Perform bench
    }
}

What I'm missing here?

[In the docs](https://benchmarkdotnet.org/articles/configs/jobs.html#run) the iteration count is controlled by `IterationCount`, `MaxIterationCount` and `MinIterationCount`. There's no explicit definition for `targetCount` — Panagiotis Kanavos, Feb 18 '22 at 12:41
@PanagiotisKanavos You are right, but, if you take a look to this example [link](https://benchmarkdotnet.org/articles/guides/choosing-run-strategy.html), you will see at the Output section that it was iterated 5 times. Also tried as it mention for the IterationCount still same behavior — semper fi, Feb 18 '22 at 13:04
For some reason, if I would change the entry point of execution to this `static void Main(string[] args) => BenchmarkRunner.Run();` It would run it properly, at once, but I would not be able to debug it. — semper fi, Feb 18 '22 at 13:12

score 4 · Accepted Answer · answered Feb 21 '22 at 12:14

The DebugInProcessConfig defines one Job:

public override IEnumerable<Job> GetJobs()
    => new[]
    {
        Job.Default

and you add one more using the attribute:

[SimpleJob(RunStrategy.ColdStart, targetCount: 1)]

so the resulting config ends up having two jobs. And BDN runs every benchmark for every job defined.

The easiest fix would be to define your own config that runs the benchmark once (Job.Dry) with the desired toolchain:

public class DebugInProcessConfigDry : DebugConfig
{
    public override IEnumerable<Job> GetJobs()
        => new[]
        {
            Job.Dry // Job.Dry instead of Job.Default
                .WithToolchain(
                    new InProcessEmitToolchain(
                        TimeSpan.FromHours(1), // 1h should be enough to debug the benchmark
                        true))
        };
}

BenchmarkDotNet running same benchmark multiple times

1 Answers1