Avoiding the overhead of C# virtual calls

Question

I have a few heavily optimized math functions that take 1-2 nanoseconds to complete. These functions are called hundreds of millions of times per second, so call overhead is a concern, despite the already-excellent performance.

In order to keep the program maintainable, the classes that provide these methods inherit an IMathFunction interface, so that other objects can directly store a specific math function and use it when needed.

public interface IMathFunction
{
  double Calculate(double input);
  double Derivate(double input);
}

public SomeObject
{
  // Note: There are cases where this is mutable
  private readonly IMathFunction mathFunction_; 

  public double SomeWork(double input, double step)
  {
    var f = mathFunction_.Calculate(input);
    var dv = mathFunction_.Derivate(input);
    return f - (dv * step);
  }
}

This interface is causing an enormous overhead compared to a direct call due to how the consuming code uses it. A direct call takes 1-2ns, whereas the virtual interface call takes 8-9ns. Evidently, the presence of the interface and its subsequent translation of the virtual call is the bottleneck for this scenario.

I would like to retain both maintainability and performance if possible. Is there a way I can resolve the virtual function to a direct call when the object is instantiated so that all subsequent calls are able to avoid the overhead? I assume this would involve creating delegates with IL, but I wouldn't know where to start with that.

Answer 1

You can cause the JIT to devirtualize your interface calls by using a struct with a constrained generic.

public SomeObject<TMathFunction> where TMathFunction: struct, IMathFunction 
{
  private readonly TMathFunction mathFunction_;

  public double SomeWork(double input, double step)
  {
    var f = mathFunction_.Calculate(input);
    var dv = mathFunction_.Derivate(input);
    return f - (dv * step);
  }
}

// ...

var obj = new SomeObject<CoolMathFunction>();
obj.SomeWork(x, y);

Here are the important pieces to note:

• The implementation of the IMathFunction interface, CoolMathFunction, is known at compile-time through a generic. This limits the applicability of this optimization quite a bit.
• A generic parameter type TMathFunction is called directly rather than the interface IMathFunction.
• The generic is constrained to implement IMathFunction so we can call those methods.
• The generic is constrained to a struct -- not strictly a requirement, but to ensure we correctly exploit how the JIT generates codes for generics: the code will still run, but we won't get the optimization we want without a struct.

When generics are instantiated, codegen is different depending on the generic parameter being a class or a struct. For classes, every instantiation actually shares the same code and is done through vtables. But structs are special: they get their own instantiation that devirtualizes the interface calls into calling the struct's methods directly, avoiding any vtables and enabling inlining.

This feature exists to avoid boxing value types into reference types every time you call a generic. It avoids allocations and is a key factor in List<T> etc. being an improvement over the non-generic List etc.

Some implementation:

I made a simple implementation of IMathFunction for testing:

class SomeImplementationByRef : IMathFunction
{
    public double Calculate(double input)
    {
        return input + input;
    }

    public double Derivate(double input)
    {
        return input * input;
    }
}

... as well as a struct version and an abstract version.

So, here's what happens with the interface version. You can see it is relatively inefficient because it performs two levels of indirection:

    return obj.SomeWork(input, step);
sub         esp,40h  
vzeroupper  
vmovaps     xmmword ptr [rsp+30h],xmm6  
vmovaps     xmmword ptr [rsp+20h],xmm7  
mov         rsi,rcx
vmovsd      qword ptr [rsp+60h],xmm2  
vmovaps     xmm6,xmm1
mov         rcx,qword ptr [rsi+8]          ; load mathFunction_ into rcx.
vmovaps     xmm1,xmm6  
mov         r11,7FFED7980020h              ; load vtable address of the IMathFunction.Calculate function.
cmp         dword ptr [rcx],ecx  
call        qword ptr [r11]                ; call IMathFunction.Calculate function which will call the actual Calculate via vtable.
vmovaps     xmm7,xmm0
mov         rcx,qword ptr [rsi+8]          ; load mathFunction_ into rcx.
vmovaps     xmm1,xmm6  
mov         r11,7FFED7980028h              ; load vtable address of the IMathFunction.Derivate function.
cmp         dword ptr [rcx],ecx  
call        qword ptr [r11]                ; call IMathFunction.Derivate function which will call the actual Derivate via vtable.
vmulsd      xmm0,xmm0,mmword ptr [rsp+60h] ; dv * step
vsubsd      xmm7,xmm7,xmm0                 ; f - (dv * step)
vmovaps     xmm0,xmm7  
vmovaps     xmm6,xmmword ptr [rsp+30h]  
vmovaps     xmm7,xmmword ptr [rsp+20h]  
add         rsp,40h  
pop         rsi  
ret  

Here's an abstract class. It's a little more efficient but only negligibly:

        return obj.SomeWork(input, step);
 sub         esp,40h  
 vzeroupper  
 vmovaps     xmmword ptr [rsp+30h],xmm6  
 vmovaps     xmmword ptr [rsp+20h],xmm7  
 mov         rsi,rcx  
 vmovsd      qword ptr [rsp+60h],xmm2  
 vmovaps     xmm6,xmm1  
 mov         rcx,qword ptr [rsi+8]           ; load mathFunction_ into rcx.
 vmovaps     xmm1,xmm6  
 mov         rax,qword ptr [rcx]             ; load object type data from mathFunction_.
 mov         rax,qword ptr [rax+40h]         ; load address of vtable into rax.
 call        qword ptr [rax+20h]             ; call Calculate via offset 0x20 of vtable.
 vmovaps     xmm7,xmm0  
 mov         rcx,qword ptr [rsi+8]           ; load mathFunction_ into rcx.
 vmovaps     xmm1,xmm6  
 mov         rax,qword ptr [rcx]             ; load object type data from mathFunction_.
 mov         rax,qword ptr [rax+40h]         ; load address of vtable into rax.
 call        qword ptr [rax+28h]             ; call Derivate via offset 0x28 of vtable.
 vmulsd      xmm0,xmm0,mmword ptr [rsp+60h]  ; dv * step
 vsubsd      xmm7,xmm7,xmm0                  ; f - (dv * step)
 vmovaps     xmm0,xmm7
 vmovaps     xmm6,xmmword ptr [rsp+30h]  
 vmovaps     xmm7,xmmword ptr [rsp+20h]  
 add         rsp,40h  
 pop         rsi  
 ret  

So both an interface and an abstract class rely heavily on branch target prediction to have acceptable performance. Even then, you can see there's quite a lot more going into it, so the best-case is still relatively slow while the worst-case is a stalled pipeline due to a mispredict.

And finally here's the generic version with a struct. You can see it's massively more efficient because everything has been fully inlined so there's no branch prediction involved. It also has the nice side effect of removing most of the stack/parameter management that was in there too, so the code becomes very compact:

    return obj.SomeWork(input, step);
push        rax  
vzeroupper  
movsx       rax,byte ptr [rcx+8]  
vmovaps     xmm0,xmm1  
vaddsd      xmm0,xmm0,xmm1  ; Calculate - got inlined
vmulsd      xmm1,xmm1,xmm1  ; Derivate - got inlined
vmulsd      xmm1,xmm1,xmm2  ; dv * step
vsubsd      xmm0,xmm0,xmm1  ; f - 
add         rsp,8  
ret  

Answer 2

I would assign the methods to delegates. This allows you to still program against the interface, while avoiding the interface method resolution.

public SomeObject
{
    private readonly Func<double, double> _calculate;
    private readonly Func<double, double> _derivate;

    public SomeObject(IMathFunction mathFunction)
    {
        _calculate = mathFunction.Calculate;
        _derivate = mathFunction.Derivate;
    }

    public double SomeWork(double input, double step)
    {
        var f = _calculate(input);
        var dv = _derivate(input);
        return f - (dv * step);
    }
}

---

In response to @CoryNelson's comment I made tests so see what the impact really is. I have sealed the function class, but this seems to make absolutely no difference since my methods are not virtual.

Test Results (mean time of 100 million iterations in ns) with the empty method time subtracted in braces:

Empty Work method: 1.48
Interface: 5.69 (4.21)
Delegates: 5.78 (4.30)
Sealed Class: 2.10 (0.62)
Class: 2.12 (0.64)

The delegate version time is about the same as for the interface version (the exact times vary from test execution to test execution). While working against the class is about 6.8 x faster (comparing times minus the empty work method time)! This means that my suggestion to work with delegates was not helpful!

What surprised me was, that I expected a much longer execution time for the interface version. Since this kind of test does not represent the exact context of the OP's code, its validity is limited.

static class TimingInterfaceVsDelegateCalls
{
    const int N = 100_000_000;
    const double msToNs = 1e6 / N;

    static SquareFunctionSealed _mathFunctionClassSealed;
    static SquareFunction _mathFunctionClass;
    static IMathFunction _mathFunctionInterface;
    static Func<double, double> _calculate;
    static Func<double, double> _derivate;

    static TimingInterfaceVsDelegateCalls()
    {
        _mathFunctionClass = new SquareFunction();
        _mathFunctionClassSealed = new SquareFunctionSealed();
        _mathFunctionInterface = _mathFunctionClassSealed;
        _calculate = _mathFunctionInterface.Calculate;
        _derivate = _mathFunctionInterface.Derivate;
    }

    interface IMathFunction
    {
        double Calculate(double input);
        double Derivate(double input);
    }

    sealed class SquareFunctionSealed : IMathFunction
    {
        public double Calculate(double input)
        {
            return input * input;
        }

        public double Derivate(double input)
        {
            return 2 * input;
        }
    }

    class SquareFunction : IMathFunction
    {
        public double Calculate(double input)
        {
            return input * input;
        }

        public double Derivate(double input)
        {
            return 2 * input;
        }
    }

    public static void Test()
    {
        var stopWatch = new Stopwatch();

        stopWatch.Start();
        for (int i = 0; i < N; i++) {
            double result = SomeWorkEmpty(i);
        }
        stopWatch.Stop();
        double emptyTime = stopWatch.ElapsedMilliseconds * msToNs;
        Console.WriteLine($"Empty Work method: {emptyTime:n2}");

        stopWatch.Restart();
        for (int i = 0; i < N; i++) {
            double result = SomeWorkInterface(i);
        }
        stopWatch.Stop();
        PrintResult("Interface", stopWatch.ElapsedMilliseconds, emptyTime);

        stopWatch.Restart();
        for (int i = 0; i < N; i++) {
            double result = SomeWorkDelegate(i);
        }
        stopWatch.Stop();
        PrintResult("Delegates", stopWatch.ElapsedMilliseconds, emptyTime);

        stopWatch.Restart();
        for (int i = 0; i < N; i++) {
            double result = SomeWorkClassSealed(i);
        }
        stopWatch.Stop();
        PrintResult("Sealed Class", stopWatch.ElapsedMilliseconds, emptyTime);

        stopWatch.Restart();
        for (int i = 0; i < N; i++) {
            double result = SomeWorkClass(i);
        }
        stopWatch.Stop();
        PrintResult("Class", stopWatch.ElapsedMilliseconds, emptyTime);
    }

    private static void PrintResult(string text, long elapsed, double emptyTime)
    {
        Console.WriteLine($"{text}: {elapsed * msToNs:n2} ({elapsed * msToNs - emptyTime:n2})");
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static double SomeWorkEmpty(int i)
    {
        return 0.0;
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static double SomeWorkInterface(int i)
    {
        double f = _mathFunctionInterface.Calculate(i);
        double dv = _mathFunctionInterface.Derivate(i);
        return f - (dv * 12.34534);
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static double SomeWorkDelegate(int i)
    {
        double f = _calculate(i);
        double dv = _derivate(i);
        return f - (dv * 12.34534);
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static double SomeWorkClassSealed(int i)
    {
        double f = _mathFunctionClassSealed.Calculate(i);
        double dv = _mathFunctionClassSealed.Derivate(i);
        return f - (dv * 12.34534);
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static double SomeWorkClass(int i)
    {
        double f = _mathFunctionClass.Calculate(i);
        double dv = _mathFunctionClass.Derivate(i);
        return f - (dv * 12.34534);
    }
}

The idea of [MethodImpl(MethodImplOptions.NoInlining)] is to prevent the compiler from calculating the addresses of the methods before the loop if the method was inlined.