Is there a way to setup each iteration of for-loop separately in benchmarks?

Question

I'm trying to run benchmarks with different number of items on the database, for example: testing response time of an endpoint for checking if the name of the item already taken with 1 item on the database, 2, 5, 10, 20, 50, 100, 200, 500, 1000. A for loop was created to add proper number of items to the database for each benchmark but the iteration setup implemented at the beginning does not work as expected and sometimes for part of the loop where 5 items should be created there are 8 instead. Is there any way to do this setup for each part of the loop separately? I know that after each test the cleanup can be used to remove items but I really want to avoid that and do all the tests while the number of items is gradually ascending and then do one clean-up at the end of the tests.

[IterationSetup]
    public void GetItemIterationSetup()
    {
        int numberOfElements = GetNumberOfElements();
        if (numberOfElements == 0 || numberOfElements == 1)
        {
            AddItemToDB(numberOfElements);
            Console.WriteLine($"Creating Item: {numberOfElements}");
        }
        else if (numberOfElements >= 2 && numberOfElements < 5)
        {
            for (int i = 0; i < 5 - numberOfElements; i++)
            {
                AddItemToDB(numberOfElements+i);
                Console.WriteLine($"Creating Item: {numberOfElements+i}");
            }
        }
        else if (numberOfElements >= 5 && numberOfElements < 10)
        {
            for (int i = 0; i < 10 - numberOfElements; i++)
            {
                AddItemToDB(numberOfElements+i);
                Console.WriteLine($"Creating Item: {numberOfElements+i}");
            }
        }
        else if (numberOfElements >= 10 && numberOfElements < 20)
        {
            for (int i = 0; i < 20 - numberOfElements; i++)
            {
                AddItemToDB(numberOfElements+i);
                Console.WriteLine($"Creating Item: {numberOfElements+i}");
            }
        }
        else if (numberOfElements >= 20 && numberOfElements < 50)
        {
            for (int i = 0; i < 50 - numberOfElements; i++)
            {
                AddItemToDB(numberOfElements+i);
                Console.WriteLine($"Creating Item: {numberOfElements + i}");
            }
        }
        else if (numberOfElements >= 50 && numberOfElements < 100)
        {
            for (int i = 0; i < 100 - numberOfElements; i++)
            {
                AddItemToDB(numberOfElements+i);
                Console.WriteLine($"Creating Item: {numberOfElements + i}");
            }
        }
        else if (numberOfElements >= 100 && numberOfElements < 200)
        {
            for (int i = 0; i < 200 - numberOfElements; i++)
            {
                AddItemToDB(numberOfElements+i);
                Console.WriteLine($"Creating Item: {numberOfElements + i}");
            }
        }
        else if (numberOfElements >= 200 && numberOfElements < 500)
        {
            for (int i = 0; i < 500 - numberOfElements; i++)
            {
                AddItemToDB(numberOfElements+i);
                Console.WriteLine($"Creating Item: {numberOfElements + i}");
            }
        }
        else if (numberOfElements >= 500 && numberOfElements < 1000)
        {
            for (int i = 0; i < 1000 - numberOfElements; i++)
            {
                AddItemToDB(numberOfElements+i);
                Console.WriteLine($"Creating Item: {numberOfElements + i}");
            }
        }
    }
    public IEnumerable<int> GetItemNumberOfElements()
    {
        yield return 1;
        yield return 2;
        yield return 5;
        yield return 10;
        yield return 20;
        yield return 50;
        yield return 100;
        yield return 200;
        yield return 500;
        yield return 1000;
    }
    [Benchmark]
    [ArgumentsSource(nameof(GetItemNumberOfElements))]
    public bool Test1(int count)
    {
        Console.WriteLine($"Proper number of Items should be: {count}, is: {GetNumberOfElements()}");
        CheckIfItemNameIsTakenUseCaseDefinition definition = CheckIfItemNameIsTakenUseCaseComposition.Definition(_telemetryClient);
        bool result = definition.Function($"Item{count}").Result;
        return result;
    }

score 0 · Answer 1 · answered Apr 08 '22 at 09:03

I am not sure if I understood correctly, but I believe that in your case you should be using Params instead of Arguments as it's applied to public field/property which can be accessed by the setup method. The disadvantage of using [Params] compared to [Arguments] is that it's applied to all benchmarks in given class.

Moreover, if the method that you are testing is not modifying the database contents, you can use a [GlobalSetup] which will be performed just once, at the beginning of benchmarking.

[Params(1, 2, 5, 10, 20, 50, 100, 200, 500, 1000)]
public int NumberOfElements;

[GlobalSetup]
public void Setup()
{
    RemoveExistingItemsFromDb(); // cleanup that needs to be implemented
    AddItemToDB(NumberOfElements);
}

[Benchmark]
public bool Test1()
{
    // dont use System.Console.WriteLine as it's expensive and might affect the reported time
    CheckIfItemNameIsTakenUseCaseDefinition definition = CheckIfItemNameIsTakenUseCaseComposition.Definition(_telemetryClient);
    return definition.Function($"Item{NumberOfElements}").Result;
}

Hi, thank you for answering! Actually I cannot add Iteration Setup to the Global Setup as the Items I am creating are modifying the database. The idea is to create specific number of items, run benchmark, add more specific number of items run another benchmark, add even more items etc. until there is 1000 items on the database - and then run Cleanup after the tests are finished. Currently my issue is that it looks like the iteration setup is being run twice, instead of once, before each benchmark test and due to that I have incorrect number of elements on the DB. — Ariet, Apr 11 '22 at 14:11

Is there a way to setup each iteration of for-loop separately in benchmarks?

1 Answers1