1

I'm stuck with the next problem. I divide my data into 10 folds. Each time, I use 1 fold as test set and the other 9 as training set (I do this ten times). On each training set, I do feature selection (filter methode with chi.squared) and then I make a SVMmodel with my training set and the selected features.
So at the end, I become 10 different models (because of the feature selection). But now I want to make a ROC-curve in R from this filter methode in general. How can I do this?

Silke

Silke
  • 177
  • 1
  • 11
  • there are serveral tutorials: http://rocr.bioinf.mpi-sb.mpg.de/ http://www.r-bloggers.com/roc-curves-and-classification/ – Verena Haunschmid Dec 03 '13 at 09:27
  • Or do you have a general problem with ROC curves? – Verena Haunschmid Dec 03 '13 at 09:27
  • No, but I want to make from this 10 different models only 1 ROC-curve that represents the performance from this feature selection method – Silke Dec 03 '13 at 09:33
  • From what I understand it only makes sense to have a curve for each model, but you can put them into one plot if you like. – Verena Haunschmid Dec 03 '13 at 09:36
  • So it is not possible to make 1 curve from this cross-validation? And has it sense to take the average from their areas under the curve? – Silke Dec 03 '13 at 09:42
  • I'm not sure about making 1 curve from 10 models, I've never done that before. But yes, you can compute the average AUC. What do you want to achieve? – Verena Haunschmid Dec 03 '13 at 09:48
  • maybe you can include some more information on your data and code. – Verena Haunschmid Dec 03 '13 at 09:50
  • I've got in my project several different methods of feature selection. And now I want a method to compare them. So I think about ROC-curves. So I search for every method the performance of that method – Silke Dec 03 '13 at 09:57
  • I see. so yes, averaging the AUC makes sense. You still should check whether the difference is significant. I'd suggest you perform a t test. You can also compute the accuracy for each 'fold' and average over the ten folds per feature selection method and perform a t test on the accuracies. Comparing the accuracies is what I have in the past, but comparing the AUC can also be useful. – Verena Haunschmid Dec 03 '13 at 10:06
  • you need to post code on your cross-validation method before anyone can help you with averaging your prediction models. – Stephen Henderson Dec 03 '13 at 11:28
  • she didn't ask for help about how to average. just if it makes sense. – Verena Haunschmid Dec 06 '13 at 07:07
  • I've found a good method. By my cross-validation I get a vector of predictions (10 little vectors become one) en this vector I can use for making my roc-curve – Silke Dec 06 '13 at 16:07

1 Answers1

2

You can indeed store the predictions if they are all on the same scale (be especially careful about this as you perform feature selection... some methods may produce scores that are dependent on the number of features) and use them to build a ROC curve. Here is the code I used for a recent paper:

library(pROC)
data(aSAH)
k <- 10
n <- dim(aSAH)[1]
indices <- sample(rep(1:k, ceiling(n/k))[1:n])

all.response <- all.predictor <- aucs <- c()
for (i in 1:k) {
  test = aSAH[indices==i,]
  learn = aSAH[indices!=i,]
  model <- glm(as.numeric(outcome)-1 ~ s100b + ndka + as.numeric(wfns), data = learn, family=binomial(link = "logit"))
  model.pred <- predict(model, newdata=test)
  aucs <- c(aucs, roc(test$outcome, model.pred)$auc)
  all.response <- c(all.response, test$outcome)
  all.predictor <- c(all.predictor, model.pred)
}

roc(all.response, all.predictor)
mean(aucs)

The roc curve is built from all.response and all.predictor that are updated at each step. This code also stores the AUC at each step in auc for comparison. Both results should be quite similar when the sample size is sufficiently large, however small samples within the cross-validation may lead to underestimated AUC as the ROC curve with all data will tend to be smoother and less underestimated by the trapezoidal rule.

Calimo
  • 7,510
  • 4
  • 39
  • 61