F# Pick function based on input?

Question

I have the following functions

let private sigmoid (z:float) =
    1.0 / (1.0 + exp(-z))
let private sigmoidM (z : Matrix<float>) : Matrix<float> =
    z.Map (fun x -> sigmoid(x))
let private sigmoidV (z:Vector<float>) =
    z.Map(fun x -> sigmoid(x))

I would like to just have sigmoid and it execute against a scalar, vector or matrix depending on the input.

That said, this function needs to be extremely performant as it is in the most critical section of loops. Any insight on how to do this? Feel free to post how with caution it will be slow if it will be slow.

Of interest: [Returning arrays of different dimensions from one function; is it possible in F#?](http://stackoverflow.com/questions/34599909/returning-arrays-of-different-dimensions-from-one-function-is-it-possible-in-f) — Guy Coder, Jun 10 '16 at 16:01
Out of curiosity why do you need the `Vector` result. When I did my neural networks I only needed the `Matrix`. The matrix only used the `sigmoid` with the matrix using the `Map` function as you did and I did in another answer. — Guy Coder, Jun 10 '16 at 16:05
Also of note when doing neural networks. The speed difference between using a CPU and GPU is so dramatic that I only use the CPU for learning and testing code, and for any serious work will only use the GPU and even then only with prewritten tested libraries, e.g. [TensorFlow](https://www.tensorflow.org/) which I used and [CNTK](https://github.com/Microsoft/CNTK) which I have not used. In short I would not worry about it if you are learning how neural networks work and getting results in less than an hour; worry about it latter when you start doing large projects. — Guy Coder, Jun 10 '16 at 16:10
Also of note: If you look at why neural networks are using the sigmoid function and how they came to choose that and other functions you will see that the heavy optimizations were done in the function analysis phase, that is why sigmoid is so useful. The optimizations were done outside of the code execution. — Guy Coder, Jun 10 '16 at 16:17
If you are using [MathNet Numerics](http://numerics.mathdotnet.com/Matrix.html) you should add that as a tag. The only [Matrix](http://numerics.mathdotnet.com/api/MathNet.Numerics.LinearAlgebra.Double/Matrix.htm) I know of that is regularly used with F# if from MathNet Numerics. — Guy Coder, Jun 10 '16 at 16:30
Of interest: [DiffSharp: Differentiable Functional Programming](http://diffsharp.github.io/DiffSharp/) This is not symbolic or numeric differenation; it is a third form which I only learned about when learning about neural networks. See: [Automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation). I learned about this when reading the [Theano](http://deeplearning.net/software/theano/tutorial/gradients.html) documentation. If you have not read the [Theano](http://deeplearning.net/software/theano/) documentation, then you should, even if you are not using it. — Guy Coder, Jun 10 '16 at 16:47

Gus · Answer 1 · 2016-06-11T14:42:24.433

You can use standard .NET overloading:

open MathNet.Numerics.LinearAlgebra

type Sigmoid() = class end with
    static member sigmoid (z:float) = 1.0 / (1.0 + exp(-z))
    static member sigmoid (z:Matrix<float>) = z.Map (fun x -> Sigmoid.sigmoid(x))
    static member sigmoid (z:Vector<float>) = z.Map (fun x -> Sigmoid.sigmoid(x))

// Usage
let x = Sigmoid.sigmoid 4.3
let y = Sigmoid.sigmoid (matrix [[1.0; 2.0]; [3.0; 4.0]])
let z = Sigmoid.sigmoid (vector  [1.0; 2.0])

// Results
val x : float = 0.9866130822
val y : Matrix<float> = 
          DenseMatrix 2x2-Double
            0.731059  0.880797
            0.952574  0.982014
val z : Vector<float> = seq [0.7310585786; 0.880797078]

This will not affect performance since the overload resolution is done at compile-time.

Not happy with standard .NET overloading? Don't want to code the function as a member? Do you want to make it more generic (accepting also float32) and extensible to other types?

Use static type constraints:

type Sigmoid() = class end with
    static member Sigmoid (_:Sigmoid, z:float  ) = 1.0  / (1.0  + exp(-z))
    static member Sigmoid (_:Sigmoid, z:float32) = 1.0f / (1.0f + exp(-z))

let inline _sigmoid (s:'Sigmoid) (x:'T) :'T =
    ((^T or ^Sigmoid) : (static member Sigmoid : 'Sigmoid * 'T -> 'T) (s, x))

let inline sigmoid x = _sigmoid (Sigmoid()) x 

type Sigmoid  with
    static member inline Sigmoid (_:Sigmoid, z:Matrix<'T>) = z.Map (fun x -> sigmoid x)
    static member inline Sigmoid (_:Sigmoid, z:Vector<'T>) = z.Map (fun x -> sigmoid x)

// Usage
let x = sigmoid 4.3
let y = sigmoid (matrix [[ 1.0; 2.0 ];[ 3.0; 4.0 ]])
let z = sigmoid (vector [ 1.0; 2.0 ])

let x' = sigmoid 4.3f
let y' = sigmoid (matrix [[1.0f; 2.0f];[ 3.0f; 4.0f]])
let z' = sigmoid (vector [ 1.0f; 2.0f])

UPDATE

Note that @TheInnerLight points out in the comments that for your specific sigmoid function you can also write:

let inline sigmoid z = 
    LanguagePrimitives.GenericOne / (LanguagePrimitives.GenericOne + exp(-z))

and that would work for float and float32

This would eventually work for vector and matrix as well, depending on their implementation.

That would be a better solution for your specific case if all operations negate, divide and exp are already generic over those types and they all support the GenericOne.

Unfortunately as of today MathNet doesn't implement GenericOne and exp for Matrix and Vector in such a way.

You don't need all the extra infrastructure around creating `Sigmoid` function that takes various numeric types. You could just write `let inline sigmoid z = LanguagePrimitives.GenericOne / (LanguagePrimitives.GenericOne + exp(-z))` — TheInnerLight, Jun 11 '16 at 11:22
@TheInnerLight Indeed for the specific case of this ``sigmoid`` function that should be the right approach, since ``1`` and ``exp`` are already generics over the desired types. So for that case is good enough. You should add it as an answer. — Gus, Jun 11 '16 at 11:40
No, it's just a minor modification of your answer, using different numeric types was just some extra information you added and not part of the original question anyway. I was just suggesting it so you could put it in your answer if you wished. — TheInnerLight, Jun 11 '16 at 11:46
Have either of you tested the version using `LanguagePrimitives` against `MathNet Numerics`? — Guy Coder, Jun 11 '16 at 12:38
@GuyCoder @TheInnerLight ``exp`` and ``GenericOne`` are not implemented, I will add a note on that. May be that would be a good enhancement for the library. Thanks. — Gus, Jun 11 '16 at 13:47
@Gustavo @GuyCoder I'd be inclined to keep the `exp` / `GenericOne` confined to the underlying numeric type and keep your implementation for the matrix/vector types. These things would be ambiguous for matrices where `GenericOne` could conceivably mean either the identity matrix or a matrix of ones and `exp` could conceivably mean point-wise or matrix exponential. — TheInnerLight, Jun 11 '16 at 13:52
@TheInnerLight That's true and that's probably the reason why it was designed that way. Anyway if you ask me since the matrix exponential is only for a square matrix I would assume the point-wise operation instead. Regarding the generic ones and zero I notice that they are already there but as a field, not a property that's why they wouldn't work for F#. — Gus, Jun 11 '16 at 14:11

F# Pick function based on input?

1 Answers1