Exception handling in BenchmarkDotNet

Question

I have a set of HTTP calls to benchmark:

public class HttpExamples
{

    [Benchmark]
    public void GetExampleCom()
    {
        var request = WebRequest.CreateHttp("http://example.com");
        var webResponse = request.GetResponse();
    }

    [Benchmark]
    public void GetExampleComSsl()
    {
        var request = WebRequest.CreateHttp("https://example.com");
        var webResponse = request.GetResponse();
    }
}

Occasionally, one of the requests will fail for some reason. At present, this seems to halt the bench-marking, what I'd prefer would be for it to handle the exception in some way.

Can this be achieved?

I'd like to:

Eliminate failures from results.
Flag that they happened

Bonus points:

Keep measures of failures so we can see how failures affect results.
Be able to see results for different types of exceptions.
Do a 'thing' on failure (clean up some what)

Umm... not to be a smart a$$, but `try/catch`? https://github.com/dotnet/BenchmarkDotNet/issues/170 seems to indicate it will work, with some caveats. — Heretic Monkey, Feb 23 '17 at 22:15
The iteration will still be included in the test runs. I'd like to have iterations ignored on error. — BanksySan, Feb 23 '17 at 22:17
The current version of BenchmarkDotNet (0.10.2) implies that a benchmark method shouldn't throw any exceptions. So, try-catch is the best possible workaround for now. Feel free to create an issue with your idea about advanced exception handling: https://github.com/dotnet/BenchmarkDotNet/issues — AndreyAkinshin, Feb 25 '17 at 15:53

score 0 · Answer 1 · answered Jan 09 '19 at 17:00

0

Try adding manual config , similar to config mentioned in https://github.com/dotnet/BenchmarkDotNet/issues/579#issuecomment-345464911

Add(JitOptimizationsValidator.DontFailOnError); // ALLOW NON-OPTIMIZED DLLS will do the trick

answered Jan 09 '19 at 17:00

mbshambharkar

386
4
13

I don't see how that would help me. This is a fix for an internal bug. – BanksySan Jan 09 '19 at 21:36
As you had mentioned:Eliminate failures from results. So adding this config will allow benchmark to continue running and in results file failed methods will contain N/A and time as 0, so you can Identify which benchmark failed. There is one method same as Setup called GlobalCleanup https://benchmarkdotnet.org/articles/features/setup-and-cleanup.html , personally I didn't find cleanup useful. – mbshambharkar Jan 13 '19 at 11:42

score 0 · Answer 2 · answered Nov 13 '20 at 11:50

BenchmarkDotNet is a micro-benchmarking harness, it was designed for a very accurate CPU-bound nano and micro benchmarks. It can do some HTTP benchmarking, but even if you use custom config you won't get as good experience as you would get with tools that were designed for HTTP benchmarking like wrk or bombardier.

Exception handling in BenchmarkDotNet

2 Answers2