Testing how learning rate affects backpropagation, Artificial neural network

Question

I have created an artificial neural network in Java that learns with a backpropagation algorithm, I have produced the following graph which shows how changing the learning rate affects the time it takes for the network to train.

It seems to show that the learning is very unstable considering it either trains correctly very quickly or gets stuck (the backpropagation will stop training at either 1 minute or a specific error threshold). I want to understand why the network is so unpredictable, is the momentum too high? do I need a adaptive learning rate? Is this a good example of how local minima affects training http://www.willamette.edu/~gorr/classes/cs449/momrate.html.

This is the graph I produced: https://i.stack.imgur.com/ooXqP.png

Are you initializing weights randomly for every experiment (for every new learning rate)? — Ibraim Ganiev, Aug 11 '15 at 18:23
Yes I create a completely fresh network, with random weights each time. — Heisenberg, Aug 11 '15 at 19:45

Ibraim Ganiev · Answer 1 · 2015-08-11T21:17:44.693

If you are initializing random weights before new experiment - you are starting optimization every time from new random point (in weight space), and for NN it's very important, because from different points with gradient descent you will converge into different local optima, ofcourse with different number of iterations and different time needed to converge. You need to generate initialization weights only once and start every experiment with new learning rate from that state, not from new random state.

Testing how learning rate affects backpropagation, Artificial neural network

1 Answers1