Algorithm not converging in given number of repetitions for neural network in R

Question

I am new to neural networks in R. I am trying to emulate the following behavior implemented using neuroph in java.

Type - Multi Layer Perceptron, Inputs - 7, Outputs - 1, Hidden - 5 neurons, Transfer Function - sigmoid, Learning Rule - Back propagation, Max error - 0.01, learning rate - 0.2

Following is the R code I implemented.

net.result <- neuralnet(Output ~ Input1 + Input2 + Input3 + Input4 + Input5 + Input6 + Input7, 
                        traindata, algorithm = "backprop", hidden = 5,
                        threshold = 0.01, learningrate = 0.2, act.fct = "logistic", 
                        linear.output = FALSE, rep =50, stepmax = 100)

The data is relatively small (120 lines) and following is a sample of the training data used. Note that the inputs are normalized and scaled between 0 and 1.

     Input1 Input2 Input3 Input4 Input5 Input6 Input7       Output
 1   0      0      0      0      0      0      0            0
 2   0      0      0      0      0      1      0.0192307692 0
 3   0      0      0      0      1      0      0.125        0
 4   0      0      0      0      1      1      0.0673076923 0
 5   0      0      0      1      0      0      0.1971153846 0
 6   0      0      0      1      0      1      0.2644230769 0.3333333333

The following is the warning I get when I execute the command mentioned above.

Warning message:
algorithm did not converge in 50 of 50 repetition(s) within the stepmax

Any clarification on why this is occurring?

score 1 · Answer 1 · answered Aug 08 '15 at 12:50

1

Increase "stepmax" from 100 to some large value in order to give the algorithm more time to converge. However, the better thing is to reduce the number of hidden nodes and then re-run the neural network

answered Aug 08 '15 at 12:50

Saurabh Kumar

149
12

I increased the stepmax to 1e5, but still it resulted in a similar warning message. When I reduced the number of hidden nodes to 3 it successfully worked. Do you know the reason why the model can't be built with a higher number of hidden neurons? – Sabra Ossen Aug 10 '15 at 02:49
Number of hidden nodes generally depends on the number of input nodes and output nodes. Since in your case, you have 7 input and 1 output nodes; hence number of hidden nodes should be less than or equal to 4. A common practice in use while selecting the number of hidden nodes is average of both input and output nodes. Hope it helps. – Saurabh Kumar Aug 10 '15 at 08:15

score 0 · Answer 2 · answered Aug 12 '15 at 08:41

You can change the number of hidden layers/hidden nodes and try different combinations through trial and error. You can try increasing the number of hidden layers in the MLP to 2. MLP with 2 hidden layers are rarely used, but they do have their uses in case of complex patterns in data. Theoretically, MLP with 2 hidden layers can be used to approximate any function.

Algorithm not converging in given number of repetitions for neural network in R

2 Answers2