0

I would like to obtain not only Accuracy and Cohen's kappa values from a k-fold cross validation, but AUC as well. I know how to obtain the avereage Accuracy, Cohen's Kappa, and AUC, as well as the Accuracy and Cohen's kappa for each fold, but I don't know how to obtain an AUC value for each fold.

Here is an example using different data

# load data
data(Sonar)

#rename data
my_data <- Sonar

#apply train control to get accuracy and cohens kappa
fitControl <-
  trainControl(
    method = "cv",
    number = 10,
    classProbs = T,
    savePredictions = T
  )

#run through k fold cross validation
model <- train(
  Class ~ .,
  data = my_data,
  method = "glm",
  trControl = fitControl
)

getTrainPerf(model)

#get every accuracy and kappa value
model$resample

I also know that I can use ROC as the metric in the train function and can fit the model to optimize ROC and can then obtain ROC values. But, I would like to optimize cohen's kappa and still see AUC scores for each fold. How might I accomplish this?

Kinz
  • 33
  • 4

0 Answers0