BenchmarkDotNet SimpleJob attribute: how so specify RuntimeMoniker equivalent to net6.0-windows

Question

I want to benchmark classes in a WPF Class Library that multi-targets both .NET 6.0 and .NET Framework 4.8. Here's project file:

WpfClassLibrary.cs:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>net6.0-windows;net4.8</TargetFrameworks>
    <UseWPF>true</UseWPF>
  </PropertyGroup>
</Project>

The class I want to test, simple instantiates a System.Windows.Documents.FlowDocument object:

namespace WpfClassLibrary
{
    using System.Windows.Documents;

    public class WpfClass
    {
        public FlowDocument Document { get; } = new FlowDocument();
    }
}

Here's the project file for the benchmarking project:

BenchmarkingWpfClassLibrary.csproj:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>net6.0-windows;net4.8</TargetFrameworks>
    <UseWPF>true</UseWPF>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.13.7" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\WpfClassLibrary\WpfClassLibrary.csproj" />
  </ItemGroup>
</Project>

And the benchmark itself:

Program.cs:

namespace Benchmarks
{
    using BenchmarkDotNet.Attributes;
    using BenchmarkDotNet.Jobs;
    using BenchmarkDotNet.Running;
    using WpfClassLibrary;

    internal class Program
    {
        static void Main(string[] args)
        {
            var results = BenchmarkRunner.Run<WpfClassBenchmarks>();
        }
    }

    //[SimpleJob(RuntimeMoniker.Net48)]
    //[SimpleJob(RuntimeMoniker.Net60)]
    public class WpfClassBenchmarks
    {
        [Benchmark]
        public void WpfClass()
        {
            _ = new WpfClass();
        }
    }
}

If I run this as-is the benchmark builds and runs fine. However, if I comment out the two SimpleJob attributes, the .NET Framework 4.8 benchmark builds and runs fine, but the .NET 6.0 benchmark does not. I get the following error:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.IO.FileNotFoundException: Could not load file or assembly 'PresentationFramework, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified.
File name: 'PresentationFramework, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
   at WpfClassLibrary.WpfClass..ctor()
   at Benchmarks.WpfClassBenchmarks.WpfClass() in C:\Users\stevensd\source\BenchmarkingWpfClassLibrary\Benchmarks\Program.cs:line 23
   at BenchmarkDotNet.Autogenerated.Runnable_0.WorkloadActionNoUnroll(Int64 invokeCount) in C:\Users\stevensd\source\BenchmarkingWpfClassLibrary\Benchmarks\bin\Release\net6.0-windows\7eaf257f-9929-493b-9995-b7827893b4db\7eaf257f-9929-493b-9995-b7827893b4db.notcs:line 311
   at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
   at BenchmarkDotNet.Engines.EngineFactory.Jit(Engine engine, Int32 jitIndex, Int32 invokeCount, Int32 unrollFactor)
   at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters)
   at BenchmarkDotNet.Autogenerated.Runnable_0.Run(IHost host, String benchmarkName) in C:\Users\stevensd\source\BenchmarkingWpfClassLibrary\Benchmarks\bin\Release\net6.0-windows\7eaf257f-9929-493b-9995-b7827893b4db\7eaf257f-9929-493b-9995-b7827893b4db.notcs:line 176
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in C:\Users\stevensd\source\BenchmarkingWpfClassLibrary\Benchmarks\bin\Release\net6.0-windows\7eaf257f-9929-493b-9995-b7827893b4db\7eaf257f-9929-493b-9995-b7827893b4db.notcs:line 57

I think what is happening is that when specifying no SimpleJob attribute, BenchmarkDotNet is building benchmark using the appropriate net6.0-windows moniker. However, when using the SimpleJob attribute with a moniker RuntimeMoniker.Net60 it's being built with the incorrect net6.0.

How can I specify a simple job that uses the runtime moniker of net60-windows and not net60?

Source code: https://github.com/DanStevens/BenchmarkingWpfClassLibrary

mm8 · Answer 1 · 2023-08-17T14:25:26.907

I think you need to use an IConfig to do this programmatically, i.e. you cannot use the SimpleJobAttribute directly:

internal class Program
{
    static void Main(string[] args)
    {
        var results = BenchmarkRunner.Run<WpfClassBenchmarks>(
            ManualConfig.Create(DefaultConfig.Instance)
                .AddJob(Job.Default.WithRuntime(ClrRuntime.Net48))
                .AddJob(Job.Default.WithToolchain(
                    CsProjCoreToolchain.From(
                        new NetCoreAppSettings(
                            targetFrameworkMoniker: "net6.0-windows",
                            runtimeFrameworkVersion: null,
                            name: "6.0")))));
    }
}

public class WpfClassBenchmarks
{
    [Benchmark]
    public void WpfClass()
    {
        _ = new WpfClass();
    }
}

Or use command-line arguments to specify the runtimes:

internal class Program
{
    static void Main(string[] args)
    {
        BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly)
            .Run(args);
    }
}

Command:

dotnet run --configuration Release --framework net4.8 --runtimes net6.0-windows net48 --filter *

score 1 · Accepted Answer · answered Aug 18 '23 at 08:25

While one can specify toolchain explicitly in the config, as suggested by mm8, if the process is ran as .NET 6.0, the default Job config will recognize and use the "net6.0-windows" moniker automatically. So the following also works:

var config = DefaultConfig.Instance
    .AddJob(Job.Default.WithId("net60"))
    .AddJob(Job.Default.WithRuntime(ClrRuntime.Net48).WithId("net48"));
var results = BenchmarkRunner.Run<WpfClassBenchmarks>(config);

Thanks Adam Sitnik for this answer.

BenchmarkDotNet SimpleJob attribute: how so specify RuntimeMoniker equivalent to net6.0-windows

2 Answers2