I would like to obtain not only Accuracy and Cohen's kappa values from a k-fold cross validation, but AUC as well. I know how to obtain the avereage Accuracy, Cohen's Kappa, and AUC, as well as the Accuracy and Cohen's kappa for each fold, but I don't know how to obtain an AUC value for each fold.
Here is an example using different data
# load data
data(Sonar)
#rename data
my_data <- Sonar
#apply train control to get accuracy and cohens kappa
fitControl <-
trainControl(
method = "cv",
number = 10,
classProbs = T,
savePredictions = T
)
#run through k fold cross validation
model <- train(
Class ~ .,
data = my_data,
method = "glm",
trControl = fitControl
)
getTrainPerf(model)
#get every accuracy and kappa value
model$resample
I also know that I can use ROC as the metric in the train function and can fit the model to optimize ROC and can then obtain ROC values. But, I would like to optimize cohen's kappa and still see AUC scores for each fold. How might I accomplish this?