How to implement a generic neural network efficiently in Haskell?

Question

A neural network is actually just a huge function with many parameters, so you might think that it would be beautiful to write such a function in a functional language, but having worked on some NN libraries for other languages, I have certain doubts about how to implement them efficiently in this paradigm.

Passing around information: If you make a graph of the dependencies of each neuron or layer, you get something like this

enter image description here

where x is the input, and f is the output. While it looks pretty straight forward in the graph, if you really treat the network as a function then f has to pass the input (plus a subset of the weight parameters) to g1 and g2, each of them has to pass these to h1, h2 and h3 who finally send the result to the layer above. Taking into account that the set of inputs plus the set of weight could be a thousand or more variables, this looks highly inefficient. In other languages you wouldn't have to do this since each neuron could preserve it's parameters, and the inputs could be passed directly to the input layer.

States: If you look at the graph, both g1 and g2 are going to separately call h2, but since h2 has to yield the same value for both, it doesn't make sense to compute its output twice. Since you don't have state's or side effects this becomes tricky, and if you have a really huge network then even with some parallel computations this is going to waste a lot of time.

Lastly, when I say that the network is generic, I mean that it can have an arbitrary shape as long as its structure doesn't contain cycles. Most libraries I've seen use a stack of layers so you just have to define the number of layer and the number of neuron in each layer, but its shape is a linear graph; this is fine for simple applications, but really hard core stuff needs networks with more complex architectures

Id like some advice in how to attack these problems since I would like to implement a library for my own needs.

Note:

I am not totally new to to language. I've used a fair amount of Functors and Monads (mostly in my FP library for C# based on haskells API), but I've never used haskell for a real application before.

Update

The State monad seems very promising!

I'll look into it. I know they use hmatrices, and don't aspire to something so generic, but maybe they also faced these problems. — Cristian Garcia, Sep 29 '14 at 03:41
If that works like a function call graph, my first instinct is that you could probably improve efficiency by using a nexus-type structure, as in this paper http://www.cs.ox.ac.uk/ralf.hinze/publications/HW03.pdf. Unfortunately, I don't know the problem well enough to give an answer though. "Hylomorphism" could also be a keyword to look up. Edit: Actually, now that I think about it, you might need more structural information than that technique would typically provide. — David Young, Sep 29 '14 at 03:41
@DavidYoung, the structure of the problem is very similar, but I'd have to carefully read the paper to see if I can apply it to this domain properly. — Cristian Garcia, Sep 29 '14 at 04:03
maybe you wouldn't mind to use some other functional language such as scala - https://github.com/yannart/Scala-Neural-Network — petrbel, Sep 29 '14 at 18:41

score 1 · Answer 1 · answered Nov 17 '14 at 18:30

I mean, since Haskell has mutual-recursion probably the easiest way to write the graph you've constructed is in terms of a big mutually-recursive where-clause:

run_graph x = run_f g1 g2 where
    g1 = run_g1 h1 h2
    g2 = run_g2 h2 h3
    h1 = run_h1 x
    h2 = run_h2 x
    h3 = run_h3 x

By giving these numbers richer types than, say, Double you can construct a sort of abstract data structure corresponding to the graph, which you could then use backpropagation on.

You might not have to worry about prematurely optimizing the graph for evaluation just yet: Haskell automatically caches the answers to functions on their inputs; this is possible because in Haskell functions are not supposed to have side-effects. There is a good chance that when your run_* functions create abstract nodes in a data structure, the eval_g1 h2 h3 will end up using the cached value from eval_h2 x that comes when you jump back to h2. If this gets too tedious then you can probably still switch to a breadth-first walk of the nodes carrying an IntMap state, before you have to explicitly start to specify layers and propagate Data.Array.Unboxed's through the layers of the graph.

How to implement a generic neural network efficiently in Haskell?

1 Answers1