caret package train: LOOCV using ROC metric

Question

I am wondering how the following code does work.

> rm(list = ls())
> data(iris)

> head(iris)
> iris$y <- factor((iris$Sepal.Length > 6), labels = c("no", "yes"))

> library(caret)
> train.control <- trainControl(method = "LOOCV", summaryFunction = twoClassSummary,
                              classProbs = T)
> knn.train <- train(y ~ . -Sepal.Length, data = iris, method = "knn",
                     preProcess = c("center", "scale"), metric = "ROC",
                     tuneGrid = expand.grid(k = 1:10),
                     trControl = train.control)

> head(knn.train$results)
  k       ROC      Sens      Spec
1 1 0.8337631 0.8314607 0.8360656
2 2 0.8702339 0.8651685 0.7540984
3 3 0.8929821 0.8314607 0.7868852
4 4 0.9187696 0.8314607 0.7868852
5 5 0.9101124 0.8202247 0.8032787
6 6 0.9067968 0.7977528 0.8524590

Explanation:

I want to use k-nearest neighbor method and find the optimal number of neighbors, k, by using the AUC as a metric.
I first load the data set "iris" and made the response variable "y".
Then, I tried to calculate the AUC for each "k" using leave-one-out cross-validation (see the 6th line (method = "LOOCV"))
In the last results, k: the number of neighbors / ROC: the area under the ROC curve for a given k / Sens: the sensitivity for the probability cutoff 0.5 for a given k / Spec: the specificity for the probability cutoff 0.5 for a given k

Here is my question. How can we calculate the AUC with LOOCV?

Think about that we may first except the first unit and then fit the model using the other units and a given k.

Next, we will try to calculate the AUC for the first unit.

But, If we have just one unit, the ROC curve may be just two points, (1,0) and (1,1) because adjusting the probability threshold, we just get the sensitivity and specificity of zero or one.

Then, I think, we cannot calculate the AUC for the first unit and then equivalently we cannot calculate the AUC using LOOCV.

But, the code above does work! Did I miss something important points?

In a sentence, I cannot understand the combination of method = "LOOCV" (in the 6th code line) and method = "knn" (in the 7th code line)

caret package train: LOOCV using ROC metric

0 Answers0