OutOfMemoryException when comparing .NET list initialized vs not initialized one using BenchmarkDotNet

Question

I want to create a benchmark showing the difference in performance between an initialized and an uninitialized .net list, but I'm having some trouble. I tried different ways of writing this benchmark but without success. Every time I run the code, I get the following:

System.OutOfMemoryException: Array dimensions exceeded supported range

ListInitializationBenchmark.cs

using BenchmarkDotNet.Attributes;

namespace list_benchmark
{
    public class ListInitializationBenchmark
    {
        private List<int> notInitializedList; // Not initialized list
        private List<int> initializedList; // Initialized list with capacity

        [Params(1000, 10000, 100000)]
        public int BatchSize { get; set; }

        [GlobalSetup]
        public void GlobalSetup()
        {
            notInitializedList = new List<int>(); // Not initialized list
            initializedList = new List<int>(BatchSize); // Initialized list with capacity
        }

        [Benchmark]
        public void ProcessNotInitializedList()
        {
            for (int i = 0; i < BatchSize; i++)
            {
                notInitializedList.Add(i);
            }
        }

        [Benchmark]
        public void ProcessInitializedList()
        {
            for (int i = 0; i < BatchSize; i++)
            {
                initializedList.Add(i);
            }
        }
    }
}

Any idea on how can I resolve this problem?

My be, you run series of test ``ProcessNotInitializedList`` , not clearing ``NotInitializedList``. Size of list=BatchSize* — ValNik, May 21 '23 at 15:19
The setup should actually be local, not global. You want fresh lists in every run. — Fildor, May 21 '23 at 15:22

score 5 · Accepted Answer · edited May 21 '23 at 15:36

BenchmarkDotNet will run your code thousands of times to get a good indication of how long it takes. However, your code uses the same instance of the list for each execution meaning it will very quickly run out of space. You need to keep the list and its initialisation inside the individual tests or change your setup to be execute per iteration. For example:

[Benchmark]
public void ProcessNotInitializedList()
{
    List<int> notInitializedList = new List<int>();

    for (int i = 0; i < BatchSize; i++)
    {
        notInitializedList.Add(i);
    }
}

[Benchmark]
public void ProcessInitializedList()
{
    List<int> initializedList = new List<int>(BatchSize);

    for (int i = 0; i < BatchSize; i++)
    {
        initializedList.Add(i);
    }
}

Or this:

[IterationSetup] // Iteration, not global setup
public void GlobalSetup()
{
    notInitializedList = new List<int>(); // Not initialized list
    initializedList = new List<int>(BatchSize); // Initialized list with capacity
}

This is absolutely correct, but it's also worth mentioning that BDN is catching OOM by default and printing information that tells the user how to proceed: https://github.com/dotnet/BenchmarkDotNet/blob/73f8fd1dc49f34b79afa4ef16e0204b1369fdd1a/src/BenchmarkDotNet/Templates/BenchmarkProgram.txt#L62-L69 — Adam Sitnik, Jun 29 '23 at 09:47

OutOfMemoryException when comparing .NET list initialized vs not initialized one using BenchmarkDotNet

1 Answers1