Automatic Differentiation in C# and F#

Question

I am having a problem getting Automatic Differentiation to work between C# and F#.

In C# I have a function that takes a double and returns a double, say:

private double Price(double yield)
{
    double price;

    price = 0;

    for (int index = 1; index <= _maturity * _frequency; index++)
    {
        price += (_coupon / _frequency) * _nominal / Math.Pow(1 + (yield / _frequency), index);
    }

    price += _nominal / Math.Pow(1 + (yield / _frequency), _maturity * _frequency);

    return price;
}

I picked this function specifically, as the Math.pow is very prohibitive, and only allows a double's or int's for its parameters.

I would like to differentiate this function using Automatic Differentiation. I have written the method for this in F#:

type Diff(d : double, df : Lazy<Diff>) = class
    member x.d = d
    member x.df = df
    static member (+) (x : Diff, y : Diff) = 
        Diff(x.d + y.d, lazy (x.df.Value + y.df.Value)) 
    static member (-) (x : Diff, y : Diff) = 
        Diff(x.d - y.d, lazy (x.df.Value - y.df.Value))
    static member (*) (x : Diff, a : double) = 
        Diff(x.d * a, lazy (x.df.Value * a))
    static member (*) (x : Diff, y : Diff) = 
        Diff(x.d * y.d, lazy ((x.df.Value * y) + (y.df.Value * x)))
    override x.ToString() =
        x.d.ToString()
end

let rec dZero = Diff(0.0, lazy dZero)

let dConst x = Diff(x, lazy dZero)

let dId x = Diff(x, lazy dConst 1.0)

let Differentiate (x:Diff) = x.df.Value

// Example function
let f (x:Diff) = x*x*x;

// Example usage:
// (f (dId 5)).ToString = "125"
// (Differentiate (f (dId 5))).ToString = "75"
// (Differentiate (Differentate (f (dId 5)))).ToString = "30"

Unfortunately, I need to feed a type Diff into my Price(..) function to produce a type Diff, which then gets fed into my Differente(..) function to return another type Diff.

My C# function however works solely on doubles (and I would like it to stay this way, as it is used in other places in my C# program).

The only way I can think to solve this is to write every function twice, which is obviously awful as:

1) I may as well just write a differentiated version each time 2) This isn't a very expandable model

So is there any way I can get around this, or perhaps coerce my double functions into Diff functions (preferably in F#). Ideally I would just like to throw a (double -> double) function in and get a Diff.ToString() out.

Sorry if this totally vague or impossible to understand. I will answer any questions in comments if this is unclear.

I hope there is a solution for this! Thanks in advance,

Ashley

Answer 1

There's no way to use your existing C# function, nor is there any easy way to lift it to a function that could operate on members of type Diff. Once the function has been compiled it is opaque and the internal structure is unavaliable; all you can do is call the function with a double argument and get a double result. Furthermore, your Price method uses operations which you haven't even defined on your Diff class anyway ((\) and Pow).

I'm not sure if it would be acceptable for your purposes, but one possible alternative would be to write a generic inline version of your Price function in F#, which could then operate on either doubles or Diffs (assuming that you add the (\) and Pow operators).

Answer 2

You can re-invent Haskell Type Classes:

interface Eq<T>
{
    bool Equal(T a, T b);
    bool NotEqual(T a, T b);
}

interface Num<T> : Eq<T>
{
    T Zero { get; }
    T Add(T a, T b);
    T Subtract(T a, T b);
    T Multiply(T a, T b);
    T Negate(T a);
}

sealed class Int : Num<int>
{
    public static readonly Int Instance = new Int();
    private Int() { }
    public bool Equal(int a, int b) { return a == b; }
    public bool NotEqual(int a, int b) { return a != b; }
    public int Zero { get { return 0; } }
    public int Add(int a, int b) { return a + b; }
    public int Subtract(int a, int b) { return a - b; }
    public int Multiply(int a, int b) { return a * b; }
    public int Negate(int a) { return -a; }
}

Then you can do:

static T F<M, T>(M m, T x) where M : Num<T>
{
    return m.Multiply(x, m.Multiply(x, x));
}

static void Main(string[] args)
{
    Console.WriteLine(F(Int.Instance, 5));  // prints "125"
}

And then with:

class Diff
{
    public readonly double d;
    public readonly Lazy<Diff> df;

    public Diff(double d, Lazy<Diff> df)
    {
        this.d = d;
        this.df = df;
    }
}

class DiffClass : Floating<Diff>
{
    public static readonly DiffClass Instance = new DiffClass();
    private static readonly Diff zero = new Diff(0.0, new Lazy<Diff>(() => DiffClass.zero));
    private DiffClass() { }
    public Diff Zero { get { return zero; } }
    public Diff Add(Diff a, Diff b) { return new Diff(a.d + b.d, new Lazy<Diff>(() => Add(a.df.Value, b.df.Value))); }
    public Diff Subtract(Diff a, Diff b) { return new Diff(a.d - b.d, new Lazy<Diff>(() => Subtract(a.df.Value, b.df.Value))); }
    public Diff Multiply(Diff a, Diff b) { return new Diff(a.d * b.d, new Lazy<Diff>(() => Add(Multiply(a.df.Value, b), Multiply(b.df.Value, a)))); }
    ...
}

You can do this:

static T Price<M, T>(M m, T _maturity, T _frequency, T _coupon, T _nominal, T yield) where M : Floating<T>
{
    T price;

    price = m.Zero;

    for (T index = m.Succ(m.Zero); m.Compare(index, m.Multiply(_maturity, _frequency)) <= 0; index = m.Succ(index))
    {
        price = m.Add(price, m.Divide(m.Multiply(m.Divide(_coupon, _frequency), _nominal), m.Power(m.Add(m.Succ(m.Zero), m.Divide(yield, _frequency)), index)));
    }

    price = m.Add(price, m.Divide(_nominal, m.Power(m.Add(m.Succ(m.Zero), m.Divide(yield, _frequency)), m.Multiply(_maturity, _frequency))));

    return price;
}

But that's not really pretty.

In fact, it almost reads like code that creates a LINQ Expression Tree. Maybe you can use Source code Expression tree transformation instead of Operator overloading to achieve Automatic differentiation?

Answer 3

The previous answers with "re-implementing type classes" are total overkill if you just need simple automatic differentiation. Use operator overloading and reify the derivatives as an array, as I show in this project. The core type you need is just:

public readonly struct Number
{
    public readonly double Magnitude;
    public readonly double[] Derivatives;

    internal Number(double m, params double[] d)
    {
        this.Magnitude = m;
        this.Derivatives = d;
    }
}

Then implement the operator translations shown on Wikipedia, so each operator on Number also operates on the Derivatives array.

You need to define your functions as operating on this Number type, but by defining the full suite of arithmetic operators on Number, this is usually just changing the parameter types, and changing any calls to the static Math.X functions to a corresponding Number.X function, ie. Math.Sin(x) -> x.Sin().

A null/empty array would mostly be operating on raw doubles, so it could be pretty close to the speed of the original code.