5

I just created a VERY large neural net, albeit on very powerful hardware, and imagine my shock and disappointment, when I realized that NeuralFit[] from NeuralNetworks` package only seems to use one core, and not even to its fullest capacity. I was heartbroken. Do I really have to write an entire NN implementation from scratch? Or did I miss something simple?

My net took 200 inputs to 2 hidden layers of 300 neurons to produce 100 outputs. I understand we're talking about trillions of calculations, but as long as I know my hardware is the weak point - that can be upgraded. It should handle training of such a net fairly well if left alone for a while (4Ghz 8-thread machine with 24Gb of 2000Mhz CL7 memory running RAID-0 SSD drives on SATA-III - I'm fairly sure).

Ideas? Suggestions? Thanks in advance for your input.

Gregory Klopper
  • 2,285
  • 1
  • 14
  • 14
  • 1
    "shock", "disappointment", "heartbroken". Hmmm, sounds a bit like an over-the-top reaction if you ask me. Anyway, I wonder whether a NN can be parallelized easily. With all those interconnections one would probably need a lot of communication between the cores which would have a considerable impact on speed. – Sjoerd C. de Vries Jan 07 '12 at 23:29
  • You might want to read this blog (http://textanddatamining.blogspot.com/2011/11/buy-or-build-practical-example-to.html) on the implementation of a NN with Mathematica – Sjoerd C. de Vries Jan 08 '12 at 00:04
  • 5
    "sounds a bit like an over-the-top reaction if you ask me" - if you pay money for a package from a company with proven track record of VERY high end mathematical solutions, and then you throw a real world problem that cannot do a single iteration of training in 24 hours, yes, I'd say that's a good reason to get a bit emotional. :-) Besides, I have a passion for what I do, it's not all just numbers for me, and this is not my job, this is my personal "child". – Gregory Klopper Jan 08 '12 at 05:04
  • I appreciate the link, I'll definitely read it, and if implementing my own is the way to go, I'll approach it seriously and do it as optimally as I can. I can also implement it in C, not in Mathematica. The whole point of using a package is to AVOID implementing my own, if someone has already done the optimization for you. In this particular package, it appears that performance was not what was optimized, which is fine, just wish it was better disclosed "on the label". – Gregory Klopper Jan 08 '12 at 05:07
  • 2
    "wonder whether a NN can be parallelized easily" - NNs are some of the easiest algorithms to parallelize because they involve a lot of independent summations/derivations, where results of individual operations do not have to be shared between branches, and can be blocked into independent sets which can be given to multiples cores, and even remote kernels. This is how all the super-massive ANNs are done. How do you think IBM did their 1.6x10^9 neuron network? On a single core? – Gregory Klopper Jan 08 '12 at 05:10
  • Multi-core NN's use networks that are specially adapted to this architecture. They work with domains or columns to restrict inter-core communication and don't do an all-to-all mapping. Is your network optimized in this way? – Sjoerd C. de Vries Jan 08 '12 at 16:20

3 Answers3

2

I am the author of the Neural Network Package. It is easy to parallelize the evaluation of a neural network given the input. That is, to compute the output of the network given the inputs (and all the weights, the parameters of the network). However, this evaluation is not very time consuming and it is not very interesting to parallellize it for most problems. On the other hand, the training of the network is often time consuming and, unfortunately, not easy to parallelize. The training can be done with a different algorithms and best ones are not easy to parallelize. My contact info can be found at the product's homepage on the Wolfram web. Improvement suggestions are very welcome.

The last version of the package works fine one version 9 and 10 if you switch off the suggestion bar (under preferences). The reason for that is that the package use the old HelpBrowser for the documentation and it crash in combination with the suggestion bar.

yours Jonas

Rojo
  • 266
  • 3
  • 15
Jonas Sjöberg
  • 31
  • 2
1

You can contact the author of the package directly, he is a very approachable fellow and might be able to make some suggestions.

berniethejet
  • 236
  • 2
  • 11
  • +1 if you could provide a little more details in your response – Sjoerd C. de Vries Jan 07 '12 at 23:16
  • @berniethejet: do you have contact info? The package is sold by wolfram without reference to the author. I'd love to reach out to the author and discuss some changes and additions to the package with him. Even willing to contribute the code or time to code. – Gregory Klopper Jan 08 '12 at 05:11
  • @Gregory: The author's name is Jonas Sjöberg. You are right, he doesn't advertise his ownership of the package. He presented at a conference a few years ago: http://library.wolfram.com/infocenter/Conferences/5417/ – berniethejet Jan 08 '12 at 15:31
  • Thanks. I'll see if I can reach him. I f anything useful come of it, I'll update here. – Gregory Klopper Jan 08 '12 at 19:54
  • Author can be find here `jonas dot sjoberg at chalmers dot se` – Dr. belisarius Jun 06 '12 at 12:24
0

I'm not sure how you wrote the code or how it is written inside the package you are using; try to use vectorization, it really speeds up the linear algebra computations. In the ml-class.org course you can see how it's made.

lmsasu
  • 7,459
  • 18
  • 79
  • 113