I know it is a strange problem and I cannot even give a MRE but with hope that someone else have encountered the similar and solved it, I am posting the question.
I have been using Rattle with all available models. I wanted to use the nnet standalone, to examine the results myself and to have more control over the parameters.
At some point I turned the entropy switch on (I do not have that many variables and factors) which resulted in error. I turned the switch back to default (FALSE) but since then nnet never worked properly. Even inside Rattle, it does not work on the training set that I used perhaps 100 times. I was able to get the probability scores, now Rattle gives me the probability scores with all other models but not with nnet. Nnet still produces binary output, even with the 'Probability' is chosen on Rattle Score options.
Has anyone encountered a similar problem with nnet?
Edit: The code part is:
mnnet <- nnet(response1 ~ ., data=dftr, size=10, skip=TRUE, MaxNWts=100000, trace=FALSE, maxit=100)
Increasing MaxNWts seems to be solving the error of too many weights
but now is taking ages to run and consumes a lot of memory. With the same dataset, I was able to run many training size possibilities within a loop which took less than a minute. (I am on a 16GB Mac Book Pro)We are talking about a set of ~10000 rows and about 20 features.
Example Code of what worked perfectly before:
pldata <- data.frame(nobs = NA,
ntrain = NA,
ntest = NA,
error = NA
)
#
obssize <- seq(0.8, 1.0, 0.1)
trsize <- seq(0.5, 0.8, 0.05)
#
set.seed(1234)
for (i in obssize){
ind1 <- sample(nrow(df), i * nrow(df))
dfp <- df[ind1,]
size <- nrow(dfp)
for (j in trsize){
ind2 <- sample(nrow(dfp), j * nrow(dfp))
dftr <- dfp[ind2, ]
dftest <- dfp[-ind2, ]
mnnet <- nnet(response1 ~ ., data=dftr, size=20, skip=TRUE, MaxNWts=10000, trace=FALSE, maxit=100)
nnetp <- predict(mnnet, newdata = dftest)
nnetp <- factor(nnetp, levels = levels(dftr$response1))
cfm <- table(nnetp, dftest$response1)
mcrate <- (length(nnetp) - sum(diag(cfm))) / length(nnetp)
trainsize <- nrow(dftr)
testsize <- nrow(dftest)
temp <- data.frame(nobs = size, ntrain = trainsize, ntest = testsize, error = mcrate)
pldata <- rbind(pldata, temp)
}
}
#
pldata <- pldata[2:nrow(pldata),]
Now the model building part is not working, let alone putting it in a loop.
Edit 2: Checked the parameters and posting what happens:
mnnet <- nnet(response1 ~ ., data=dftr, size=10, skip=TRUE, MaxNWts=100000, trace=FALSE, maxit=100, entropy = FALSE)
> mnnet$entropy
[1] TRUE
Once the entropy is set to 'TRUE' it seems to be impossible to set it back to default.