Can benchmarks be created dynamically in BenchmarkDotNet?

Question

When writing unit tests, we can add more test cases by simply adding elements to a collection, for example using TestCaseSource in NUnit.

Is it possible to do something similar in BenchmarkDotNet, and create a set of benchmarks from a collection?

This would save a lot of boilerplate, especially when benchmarking multiple combinations of inputs or doing additional testing of the benchmarked methods.

Minimal example code:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

public interface IAlgorithm
{
    public void DoWork();
}

public class AlgorithmA: IAlgorithm
{
    public void DoWork() { } // something slow
}

public class AlgorithmB : IAlgorithm
{
    public void DoWork() { } // something slow
}

public class MyBenchmarks
{
    AlgorithmA classA = new AlgorithmA();
    AlgorithmB classB = new AlgorithmB();

    [Benchmark]
    public void A() => classA.DoWork();

    [Benchmark]
    public void B() => classB.DoWork();
}

public class WhatIWouldLike
{
    IAlgorithm[] classes = new IAlgorithm[] { new AlgorithmA(), new AlgorithmB() };

    // ...automatically create a benchmark for DoWork() on element of the array
}

class Program
{
    static void Main(string[] args)
    {
        BenchmarkRunner.Run<MyBenchmarks>();
    }
}

Sweeper · Answer 1 · 2022-02-22T18:17:46.877

There are parameterised benchmarks. In this case you can use a ArgumentsSource or ParamsSource.

Example for ArgumentsSource, which is similar to TestCaseSource in terms of how you'd use it:

public class Algorithm1: IAlgorithm {
    public void DoWork() {
        ...
    }

    // override ToString so that we have a more readable table of results
    public override string ToString() => "Algorithm 1";
}

public class Algorithm2: IAlgorithm {
    public void DoWork() {
        ...
    }
    public override string ToString() => "Algorithm 2";
}

public class MyBenchmarks {
    public IEnumerable<IAlgorithm> Algorithms() {
        yield return new Algorithm1();
        yield return new Algorithm2();
        // add more algorithms here if needed
    }

    [Benchmark]
    [ArgumentsSource(nameof(Algorithms))]
    public void RunAlgorithm(IAlgorithm algorithm) => algorithm.DoWork();
}

You can get a table such as:

|       Method |   algorithm |     Mean |     Error |    StdDev |
|------------- |------------ |---------:|----------:|----------:|
| RunAlgorithm | Algorithm 1 | 1.146 ms | 0.0065 ms | 0.0061 ms |
| RunAlgorithm | Algorithm 2 | 2.223 ms | 0.0120 ms | 0.0106 ms |

Can benchmarks be created dynamically in BenchmarkDotNet?

1 Answers1