DI containers leak memory or BenchmarksDotNet MemoryDiagnoser delivers inaccurate measurements?

Question

Introduction

We are trying to catch potential memory leaks using BenchmarksDotNet.

For the simplicity of example, here is an unsophisticated TestClass:

public class TestClass 
{
    private readonly string _eventName;

    public TestClass(string eventName)
    {
        _eventName = eventName;
    }

    public void TestMethod() =>
        Console.Write($@"{_eventName} ");
}

We are implementing benchmarking though NUnit tests in netcoreapp2.0:

[TestFixture]
[MemoryDiagnoser]
public class TestBenchmarks
{
    [Test]
    public void RunTestBenchmarks() =>
        BenchmarkRunner.Run<TestBenchmarks>(new BenchmarksConfig());

    [Benchmark]
    public void TestBenchmark1() =>
        CreateTestClass("Test");

    private void CreateTestClass(string eventName)
    {
        var testClass = new TestClass(eventName);
        testClass.TestMethod();
    }
}

The test output contains following summary:

         Method | Mean | Error | Allocated |
--------------- |-----:|------:|----------:|
 TestBenchmark1 |   NA |    NA |       0 B |

The test output also contains all the Console.Write output which proves that 0 B here means no memory was leaked rather than no code was run because of compiler optimization.

Problem

The confusion begins when we attempt to resolve TestClass with TinyIoC container:

[TestFixture]
[MemoryDiagnoser]
public class TestBenchmarks
{
    private TinyIoCContainer _container;

    [GlobalSetup]
    public void SetUp() =>
        _container = TinyIoCContainer.Current;

    [Test]
    public void RunTestBenchmarks() =>
        BenchmarkRunner.Run<TestBenchmarks>(new BenchmarksConfig());

    [Benchmark]
    public void TestBenchmark1() => 
        ResolveTestClass("Test");

    private void ResolveTestClass(string eventName)
    {
        var testClass = _container.Resolve<TestClass>(
            NamedParameterOverloads.FromIDictionary(
                new Dictionary<string, object> {["eventName"] = eventName}));
        testClass.TestMethod();
    }
}

The summary indicates 1.07 KB was leaked.

         Method | Mean | Error | Allocated |
--------------- |-----:|------:|----------:|
 TestBenchmark1 |   NA |    NA |   1.07 KB |

Allocated value increases proportionally to the number of ResolveTestClass calls from TestBenchmark1, the summary for

[Benchmark]
public void TestBenchmark1() 
{
    ResolveTestClass("Test");
    ResolveTestClass("Test");
}

is

         Method | Mean | Error | Allocated |
--------------- |-----:|------:|----------:|
 TestBenchmark1 |   NA |    NA |   2.14 KB |

This indicates that either TinyIoC is keeping the reference to each resolved object (which does not seem to be true according to source code) or BenchmarksDotNet measurements include some additional memory allocations outside of method marked with [Benchmark] attribute.

The config used in both cases:

public class BenchmarksConfig : ManualConfig
{
    public BenchmarksConfig()
    {
        Add(JitOptimizationsValidator.DontFailOnError); 

        Add(DefaultConfig.Instance.GetLoggers().ToArray()); 
        Add(DefaultConfig.Instance.GetColumnProviders().ToArray()); 

        Add(Job.Default
            .WithLaunchCount(1)
            .WithTargetCount(1)
            .WithWarmupCount(1)
            .WithInvocationCount(16));

        Add(MemoryDiagnoser.Default);
    }
}

By the way, replacing TinyIoC with Autofac dependency injection framework didn't change the situation much.

Questions

Does it mean all DI framework have to implement some sort of cache for resolved objects? Does it mean BenchmarksDotNet is used in wrong way in a given example? Is it a good idea to hunt for memory leaks with the combination of NUnit and BenchmarksDotNet in the first place?

Answer 1

I am the person who implemented MemoryDiagnoser for BenchmarkDotNet and I am very happy to answer this question.

But first I am going to describe how the MemoryDiagnoser works.

1. It gets the number of allocated memory by using available API.
2. It performs one extra iteration of benchmark runs. In your case, it's 16 (.WithInvocationCount(16))
3. It gets the number of allocated memory by using available API.

final result = (totalMemoryAfter - totalMemoryBefore) / invocationCount

How accurate is the result? It is as accurate as the available APIs that we are using: GC.GetAllocatedBytesForCurrentThread() for .NET Core 1.1+ and AppDomain.MonitoringTotalAllocatedMemorySize for .NET 4.6+.

The thing called GC Allocation Quantum defines the size of allocated memory. It is usually 8k bytes.

What does it really mean: if we allocate a single object with new object() and GC needs to allocate memory for it (the current segment is full) it's going to allocate 8k of memory. And both APIs are going to report 8k memory allocated after a single object allocation.

Console.WriteLine(AppDomain.MonitoringTotalAllocatedMemorySize);
GC.KeepAlive(new object());
Console.WriteLine(AppDomain.MonitoringTotalAllocatedMemorySize);

might end up in reporting:

x
x + 8000

How BenchmarkDotNet deals with this problem? We perform a LOT of invocations (usually millions or billions), so minimize the allocation quantum size problem (it's never 8k for us).

How to fix the problem in your case: set the WithInvocationCount to a bigger number (maybe 1000).

To verify the results you might consider using some Memory Profiler. Personally I used Visual Studio Memory Profiler which is part of Visual Studio.

Another alternative is to use JetBrains.DotMemoryUnit. It's most probably the best tool in your case.