How can I implement a R code for AUC in genetic algorithm

Question

I am doing research on applying a genetic algorithm to binary logistic regression. I have a few questions to be clarified. Can you please help me?

Can I use AIC or BIC as the fitness function in the GA? (I used them and results show that GA is more accurate than the traditional binary logistic model. However, I found that in most of the papers, they used AUC as the fitness function)
I tried GA using AUC as the fitness function according to this paper (http://atm.amegroups.com/article/view/18292/html) and it gives following error. Can you create a small reproducible example to overcome this problem?

Error in model.frame.default(formula = as.numeric(tey) ~ predict.glm(trm, : variable lengths differ (found for 'predict.glm(trm, newdata = ted, type = "response")')

score 0 · Answer 1 · answered Apr 08 '20 at 09:04

0

In the galgo package, the cost function can be custom-defined. Can you run the program as described in the paper? For example, you can define AUC as your goal; and you use neural network for the prediction the following code can help:

reg.fitness <- function(chr, parent,tr,te,res) {
  try <- as.factor(parent$data$classes[tr])
  trd <-
    data.frame(parent$data$data[tr,as.numeric(chr)])
  trm <- nnet::nnet(try ~ ., data = cbind(trd,try=try),trace=F,
                    size = 5)
  tey <- as.factor(parent$data$classes[te])
  ted <-
    data.frame(parent$data$data[te,as.numeric(chr)])
  pred

 <- predict(trm,newdata = cbind(ted,tey=tey),type = "raw")
  if(res){
    roc(tey,pred,
        levels=levels(tey),
        direction = "<")$auc
  }
  else{
    predict(trm,newdata=cbind(ted,tey=tey),type="class")
  }

}

you can adapt the model by modify this chunk: trm <- nnet::nnet(try ~ ., data = cbind(trd,try=try),trace=F, size = 5)

answered Apr 08 '20 at 09:04

Z. Zhang

637
4
16

@ChathuriAththanayake can you rebuild your fitnesses function with above code? – Z. Zhang Apr 08 '20 at 13:58
Thank you, @Z. Zhang I got this error (**Error in model.frame.default(formula = as.numeric(tey) ~ predict.glm(trm, : variable lengths differ (found for 'predict.glm(trm, newdata = ted, type = "response")')**) when I'm running the code according to the paper. However, I couldn't find the object 'pred' in the code for the neural network for the prediction. How can I adapt it? – Chathuri Aththanayake Apr 08 '20 at 14:03
I couldn't update the object called 'pred' in the above code. – Chathuri Aththanayake Apr 08 '20 at 14:07
@ Z. Zhang. Thank you very much. I was able to run the code. Unfortunately, the fitness value is 1 from the beginning. I tried this code for a different data set, it also found that the fitness function = 1 from the 1st iteration. This is the full code that I used for 3 independent categorical variables with binary dependent variable. – Chathuri Aththanayake Apr 10 '20 at 06:45
What’s the sample size ? – Z. Zhang Apr 10 '20 at 06:52
@ Z. Zhang total sample size = 498 and I took 331 for training – Chathuri Aththanayake Apr 10 '20 at 07:19

Chathuri Aththanayake · Answer 2 · 2020-04-10T07:41:33.430

@Z. Zhang

reg.fitness = function(chr, parent,tr,te,res) {
  try=as.factor(parent$data$classes[tr])
  trd =
    data.frame(parent$data$data[tr,as.numeric(chr)])
  trm = nnet::nnet(try ~ ., data = cbind(trd,try=try),trace=F,
                    size = 3)
  tey = as.factor(parent$data$classes[te])
  ted =
    data.frame(parent$data$data[te,as.numeric(chr)])
 pred=predict(trm,newdata = cbind(ted,tey=tey),type = "raw")
  if(res){
    roc(tey,pred,levels=levels(tey),
        direction = "<")$auc
  }
  else{
    predict(trm,newdata=cbind(ted,tey=tey),type="class")
  }

}




reg.bb = configBB.VarSel(data=t(data_set[,-ncol(data_set)]), 
                          classes=data_set$y, 
                          classification.method="user", 
                          classification.userFitnessFunc=reg.fitness, 
                          chromosomeSize=3, niches=1, maxSolutions=10,
                          goalFitness = 0.9, saveVariable="reg.bb",
                          saveFrequency=50, saveFile="reg.bb.Rdata", 
                          main="Logistic")
blast(reg.bb)

This is the output that I got for first 4 iterations

[Bb] Starting, Solutions=10
[Bb]    #bb Sol Last    Fitness %Fit    Gen Time    Elapsed Total   Remaining

[e] Starting: Fitness Goal=0.9, Generations=(10 : 200)
[e] Elapsed Time    Generation  Fitness %Fit    [Next Generations]
[e] 0h 0m 0s    (m) 0   1   111.11% +GGGGGGGGGG
[e] 0h 0m 9s    *** 11  1   111.11% FINISH: 1 2 1 

[Bb]    1   1   Sol Ok  1   111.11% 11  9.33s   9s  10s 42s (0h 0m 42s )

[e] Starting: Fitness Goal=0.9, Generations=(10 : 200)
[e] Elapsed Time    Generation  Fitness %Fit    [Next Generations]
[e] 0h 0m 0s    (m) 0   1   111.11% +GGGGGGGGGG
[e] 0h 0m 10s   *** 11  1   111.11% FINISH: 1 1 3 

[Bb]    2   2   Sol Ok  1   111.11% 11  10.35s  20s 22s 50s (0h 0m 50s )

[e] Starting: Fitness Goal=0.9, Generations=(10 : 200)
[e] Elapsed Time    Generation  Fitness %Fit    [Next Generations]
[e] 0h 0m 0s    (m) 0   1   111.11% +GGGGGGGGGG
[e] 0h 0m 10s   *** 11  1   111.11% FINISH: 3 1 1 

[Bb]    3   3   Sol Ok  1   111.11% 11  9.93s   30s 34s 50s (0h 0m 50s )

[e] Starting: Fitness Goal=0.9, Generations=(10 : 200)
[e] Elapsed Time    Generation  Fitness %Fit    [Next Generations]
[e] 0h 0m 0s    (m) 0   1   111.11% +GGGGGGGGGG
[e] 0h 0m 10s   *** 11  1   111.11% FINISH: 1 2 2 

[Bb]    4   4   Sol Ok  1   111.11% 11  10s 40s 45s 45s (0h 0m 45)

Plot for all 1000 iterations where it gave same fitness function value as 1

I cannot reproduce your error; can you post a simulated example？ — Z. Zhang, Apr 10 '20 at 07:21
@ Z. Zhang , Here I posted output for first 4 iterations and plot for 1000 iterations which gave the same fitness value — Chathuri Aththanayake, Apr 10 '20 at 07:43
A reproducible example means what can also run on my computer — Z. Zhang, Apr 11 '20 at 22:36
@Z.Zhang, I tried the code for several data sets. But all of them gave the same fitness value as 1. I posted my code as a reproducible example. Can you please help me to clarify the fault — Chathuri Aththanayake, Apr 18 '20 at 17:02

score 0 · Answer 3 · answered Apr 13 '20 at 05:59

@ Z. Zhang, here is a reproducible example of my code.

library(pROC)
library(galgo) 
library(rtkore)
library(Rcpp)
library(aod)
mydata <- read.csv("https://stats.idre.ucla.edu/stat/data/binary.csv")
attach(mydata)


reg.fitness = function(chr, parent,tr,te,res) {
  try=as.factor(parent$data$classes[tr])
  trd =
    data.frame(parent$data$data[tr,as.numeric(chr)])
  trm = nnet::nnet(try ~ ., data = cbind(trd,try=try),trace=F,size = 2)
  tey = as.factor(parent$data$classes[te])
  ted =
    data.frame(parent$data$data[te,as.numeric(chr)])
 pred=predict(trm,newdata = cbind(ted,tey=tey),type = "raw")
  if(res){
    roc(tey,pred,levels=levels(tey),
        direction = "<")$auc
  }
  else{
    predict(trm,newdata=cbind(ted,tey=tey),type="class")
  }

}




reg.bb = configBB.VarSel(data=t(mydata[,-ncol(mydata)]), 
                          classes=admit ,
                          classification.method="user", 
                          classification.userFitnessFunc=reg.fitness, 
                          chromosomeSize=2 ,niches=1, maxSolutions=10,
                          goalFitness = 0.9, saveVariable="reg.bb",
                          saveFrequency=50, saveFile="reg.bb.Rdata", 
                          main="Logistic")
blast(reg.bb)

Plot(blast(reg.bb))

score 0 · Answer 4 · answered Apr 18 '20 at 23:28

The problem is that you use the admin to predict admin, which will of course get a AUC of 1. here is the revised code which can run expectedly in my computer.

library(pROC) library(galgo)
library(rtkore) library(Rcpp) library(aod) mydata <- read.csv("https://stats.idre.ucla.edu/stat/data/binary.csv")

reg.fitness = function(chr, parent,tr,te,res) {
try=as.factor(parent$data$classes[tr]) trd = data.frame(parent$data$data[tr,as.numeric(chr)]) trm = nnet::nnet(try ~ ., data = cbind(trd,try=try),trace=F,size = 2) tey = as.factor(parent$data$classes[te]) ted = data.frame(parent$data$data[te,as.numeric(chr)]) pred=predict(trm,newdata = cbind(ted,tey=tey),type = "raw") if(res){ roc(tey,pred,levels=levels(tey), direction = "<")$auc } else{ predict(trm,newdata=cbind(ted,tey=tey),type="class") } }

reg.bb = configBB.VarSel(data=t(mydata[,-1]), classes=mydata$admit , classification.method="user", classification.userFitnessFunc=reg.fitness, chromosomeSize=2 ,niches=1, maxSolutions=10, goalFitness = 0.9, saveVariable="reg.bb", saveFrequency=50, saveFile="reg.bb.Rdata", main="Logistic")

blast(reg.bb)

Plot(blast(reg.bb))

Thank you very much. It works well and I was able to apply this for my data set as well. Cheers!!! — Chathuri Aththanayake, Apr 19 '20 at 07:08
@ Z. Zhang, I did a comparison between two GAs for my study. For one method I adopted your R code. It will be really grateful If you can give some idea about ANN part in your code. For example, Inputs, outputs and how many hidden layers are there? Thank you in advance. — Chathuri Aththanayake, Jun 28 '20 at 18:19

score 0 · Answer 5 · answered Jun 28 '20 at 22:04

0

for the ANN in the example, I use the nnet function, it has only one hidden layer, the input and output is determined by the data dimension. You can reference the manual help for the nnet function. Furthermore you can apply any structure of ANN by replace the function with any other functions

answered Jun 28 '20 at 22:04

Z. Zhang

637
4
16

Thank you very much @Z. Zhang. I found the manual help for nnet. – Chathuri Aththanayake Jul 03 '20 at 18:59

How can I implement a R code for AUC in genetic algorithm

5 Answers5