Higher order functions in AleaGPU C#

Question

I am trying to code C# versions (in C# style) of the F# reduce functions found here:

https://github.com/quantalea/AleaGPUTutorial/tree/master/src/fsharp/examples/generic_reduce

More specific to my question, take this function for example:

let multiReduce (opExpr:Expr<'T -> 'T -> 'T>) numWarps =
    let warpStride = WARP_SIZE + WARP_SIZE / 2 + 1
    let sharedSize = numwarps * warpStride

    <@ fun tid (x:'T) ->
        // stuff
    @>

I'm primarily an F# guy, and I'm not quite sure how I should go about coding functions like these in C#. For the C# version, the multiReduce function will be a class member. So if I wanted to do a more direct translation of the F# code, I would return a Func from my MultiReduce member.

The other option would be to "flatten" the multiReduce function, so that my C# member version would have two extra parameters. So...

public T MultiReduce(Func<T,T,T> op, int numWarps, int tid, T x)
{
    // stuff
}

But I don't think this would work for AleaGPU coding in all cases because the quoted expression in the F# version is a device function. You need the nested function structure to be able to separate the assignment of certain variables from the actual invocation of the function.

Another way I see to do it would be to make a MultiReduce class and have the opExpr and numWarps as fields, then make the function in the quotation a class member.

So how are higher order functions like these generally implemented in AleaGPU-C#? I don't think it's good to return Func<..> everywhere since I don't see this done much in C# coding. Is AleaGPU a special case where this would be ok?

A basic AleaGPU C# implementation looks like this:

internal class TransformModule<T> : ILGPUModule
{
    private readonly Func<T, T> op;

    public TransformModule(GPUModuleTarget target, Func<T, T> opFunc)
        : base(target)
    {
        op = opFunc;
    }

    [Kernel]
    public void Kernel(int n, deviceptr<T> x, deviceptr<T> y)
    {
        var start = blockIdx.x * blockDim.x + threadIdx.x;
        var stride = gridDim.x * blockDim.x;
        for (var i = start; i < n; i += stride)
            y[i] = op(x[i]);
    }

    public void Apply(int n, deviceptr<T> x, deviceptr<T> y)
    {
        const int blockSize = 256;
        var numSm = this.GPUWorker.Device.Attributes.MULTIPROCESSOR_COUNT;
        var gridSize = Math.Min(16 * numSm, Common.divup(n, blockSize));
        var lp = new LaunchParam(gridSize, blockSize);
        GPULaunch(Kernel, lp, n, x, y);
    }

    public T[] Apply(T[] x)
    {
        using (var dx = GPUWorker.Malloc(x))
        using (var dy = GPUWorker.Malloc<T>(x.Length))
        {
            Apply(x.Length, dx.Ptr, dy.Ptr);
            return dy.Gather();
        }
    }
}

Answer 1

Higher-order functions are not nearly as ubiquitous in C# as they are in F#. While there are plenty of examples of accepting functions as arguments, C# code rarely returns functions as results. I guess this is partly because the code comes out very ugly (Func<T,U> everywhere) and partly because C# programmers are not generally used to functional style and gravitate more toward OO ways.

In particular, there is no automatic currying/partial application in C#. You can think of it as if all your F# functions always had tupled parameters. In fact, that's how a multi-parameter C# method would look if you called it from F#.

I must also note that the function in your code is not, in fact, "higher-order". It neither accepts nor returns any functions. Instead, it accepts and returns quotations, which is not at all the same thing. Function is, roughly speaking, a reference to a piece of code, but quotation is a data structure. They look similar, but they're completely different animals.

C# does, too, have its own quotations, represented by the type System.Linq.Expressions.Expression<T> (where T must be a delegate type). However, they are not the same thing as F# quotations. From F# side, you can (sorta) use C# quotation, but not the other way around.
Both F# and C# quotations have their strengths and weaknesses. In particular, C# supports compilation, F# doesn't. F# supports splicing, C# doesn't.

Which brings me to the next point: you probably need splicing. Because you are using opExpr in the body of the returned quotation, aren't you?
And C# doesn't have an out-of-the-box support for it. Yes, it is theoretically possible to implement splicing as a library function, but for some reason there is no de-facto standard, regularly maintained implementation. We, for one, had to roll our own. It's open source, too, and pretty straightforward, so feel free to use it.

Now, having said all the above, I want to express a doubt that you would be able to use C# for this at all. I don't really know how AleaGPU works, but it looks like it expects you to return an F# quotation, which it then, presumably, compiles into GPU code. If that's the case, because C# and F# quotations are two different things, you probably won't be able to return a C# quotation to AleaGPU in lieu of the F# one. Unless it has separate support for it, of course.