R - neuralnet - Traditional backprop seems strange

Question

I am experimenting with the different algorithms in the neuralnet package but when I try the traditional backprop algorithm the results are very strange/disappointing. Almost all the calculated results are ~.33??? I assume I must be using the algorithm incorrectly as if I run it with the default rprop+ it does differentiate between samples. Surely normal backpropagation is not this bad especially if it is able to converge so quickly to the provided threshold.

library(neuralnet)
data(infert)

set.seed(123)
fit <- neuralnet::neuralnet(formula = case~age+parity+induced+spontaneous, 
                            data = infert, hidden = 3, 
                            learningrate = 0.01, 
                            algorithm =  "backprop", 
                            err.fct = "ce", 
                            linear.output = FALSE,
                            lifesign = 'full', 
                            lifesign.step = 100)

preds <- neuralnet::compute(fit, infert[,c("age","parity","induced","spontaneous")])$net.result

summary(preds)
       V1           
 Min.   :0.3347060  
 1st Qu.:0.3347158  
 Median :0.3347161  
 Mean   :0.3347158  
 3rd Qu.:0.3347162  
 Max.   :0.3347286

Are some settings supposed to be different here?

Example default neuralnet

set.seed(123)
fit <- neuralnet::neuralnet(formula = case~age+parity+induced+spontaneous, 
                            data = infert, hidden = 3, 
                            err.fct = "ce", 
                            linear.output = FALSE,
                            lifesign = 'full', 
                            lifesign.step = 100)

preds <- neuralnet::compute(fit, infert[,c("age","parity","induced","spontaneous")])$net.result

summary(preds)
       V1           
 Min.   :0.1360947  
 1st Qu.:0.1516387  
 Median :0.1984035  
 Mean   :0.3346734  
 3rd Qu.:0.4838288  
 Max.   :1.0000000

score 3 · Accepted Answer · edited May 23 '17 at 10:28

It is recommended that you normalize your data before feeding to a neural network. If you do that, then you're good to go:

library(neuralnet)
data(infert)

set.seed(123)
infert[,c('age','parity','induced','spontaneous')] <- scale(infert[,c('age','parity','induced','spontaneous')])
fit <- neuralnet::neuralnet(formula = case~age+parity+induced+spontaneous, 
                            data = infert, hidden = 3, 
                            learningrate = 0.01, 
                            algorithm =  "backprop", 
                            err.fct = "ce", 
                            linear.output = FALSE,
                            lifesign = 'full', 
                            lifesign.step = 100)

preds <- neuralnet::compute(fit, infert[,c("age","parity","induced","spontaneous")])$net.result
summary(preds)
       V1            
 Min.   :0.02138785  
 1st Qu.:0.21002456  
 Median :0.21463423  
 Mean   :0.33471568  
 3rd Qu.:0.47239818  
 Max.   :0.97874839

There are actually a few questions on SO dealing with this. Why do we have to normalize the input for an artificial neural network? seemed to have some of the most detail.

Interesting, I should have known about the scaling. Thank you. Do you have any idea why the `rprop+` algorithm is able to deal with this by default without scaling then? — cdeterman, Jul 05 '16 at 20:07
I don't - I would imagine that it's in the code somewhere that it's done by default, but I don't know why it would be different. — Tchotchke, Jul 12 '16 at 15:22
Fair enough, thank you for answering my question. I will poke around and perhaps ask that question again later. — cdeterman, Jul 12 '16 at 16:26

R - neuralnet - Traditional backprop seems strange

1 Answers1