cross-validate predictions for caret and svm

Question

There seem to be differences between the ROC/Sens/Spec that is produced when tuning the model, to the actual predictions made by the model on the same dataset. I'm using caret which uses kernlab's ksvm. I'm not experiencing this problem with glm.

data(iris)
library(caret)
iris <- subset(iris,Species == "versicolor" | Species == "setosa") # we need only two output classess
iris$noise <- runif(nrow(iris)) # add noise - otherwise the model is too "perfect"
iris$Species <- factor(iris$Species)
fitControl <- trainControl(method = "repeatedcv",number = 10, repeats = 5, savePredictions = TRUE, classProbs = TRUE, summaryFunction = twoClassSummary)

ir <- train(Species ~ Sepal.Length + noise, data=iris,method = "svmRadial", preProc = c("center", "scale"), trControl=fitControl,metric="ROC")
confusionMatrix(predict(ir), iris$Species, positive = "setosa")
getTrainperf(ir) # same as in the model summary

What is the source of this discrepancy? which ones are the "real", post-cross-validation predictions?

score 2 · Accepted Answer · answered Dec 15 '16 at 19:36

It seems the function getTrainPerf gives the mean performance results of the best tuned parameters averaged across the repeated cross validations folds.

Here is how getTrainPerf works:

getTrainPerf(ir) 
#  TrainROC TrainSens TrainSpec    method
#1   0.9096     0.844     0.884 svmRadial

which is achieved in the following way:

ir$results
#      sigma    C    ROC  Sens  Spec      ROCSD    SensSD    SpecSD
#1 0.7856182 0.25 0.9064 0.860 0.888 0.09306044 0.1355262 0.1222911
#2 0.7856182 0.50 0.9096 0.844 0.884 0.08882360 0.1473023 0.1218229
#3 0.7856182 1.00 0.8968 0.836 0.884 0.09146071 0.1495026 0.1218229
ir$bestTune
#      sigma   C
#2 0.7856182 0.5
merge(ir$results, ir$bestTune)
#      sigma   C    ROC  Sens  Spec     ROCSD    SensSD    SpecSD
#1 0.7856182 0.5 0.9096 0.844 0.884 0.0888236 0.1473023 0.1218229

which can also be obtained from the performance results on the cross validation folds (10 folds, 5 repeats, 10*5=50 total values for each of the performance measures).

colMeans(ir$resample[1:3])
#     ROC   Sens   Spec 
#  0.9096 0.8440 0.8840

Hence, getTrainPerfonly gives the summary of the cross-validation performances on the data folds held-out for validation at different times (not on the entire training dataset) with the best tuned parameters (sigma, C).

But if you want to predict on your entire training dataset, you need to use the predict function with the tuned model.

cross-validate predictions for caret and svm

1 Answers1