Calculate cutoff and sensitivity for specific values of specificity?

Question

After calculating several regression models, I want to calculate sensitivity-values and the cut-off for pre-specified values of specificity (i.e, 0.99, 0.90, 0.85 and so on) to find the best model. I have created code to calculate sensitivity and specificity for given values of the cut-off (from 0.1 to 0.9), but now I want to use specific values of specificity (i.e., calculate the corresponding cut-off value and sensitivity-values), and here I'm stuck.

Suppose I have the following regression model (using the example dataset 'mtcars'):

    data(mtcars)
    model <- glm(formula= vs ~ wt + disp, data=mtcars, family=binomial)

Here is the code I've used for the calculation of sens and spec for given values of the cut-off:

    predvalues <- model$fitted.values
getMisclass <- function(cutoff, p, labels){
  d <- cut(predvalues, breaks=c(-Inf, cutoff, Inf), labels=c("0", "1"))
  print(confusionMatrix(d, mtcars$vs, positive="1"))
  cat("cutoff", cutoff, ":\n")
  t <- table(d, mtcars$vs)
  print(round(sum(t[c(1,4)])/sum(t), 2)) 
}
cutoffs <- seq(.1,.9,by=.1)
sapply(cutoffs, getMisclass, p=predval, labels=mtcars$vs)

Can someone help me how to rewrite this code for the calculation of sensitivity and cut-off scores given a range of specificity-values? Is it possible? The values for the cutoff should be

    cutoffs <- c(0.99, 0.90, 0.85, 0.80, 0.75)

Thanks a lot!

Mmm okay, sorry for that! @geekoverdose solved my issue, so I think it's no longer necessary to edit my question... — Hed, Aug 04 '16 at 08:38

score 2 · Accepted Answer · answered Aug 03 '16 at 11:33

This is closely related to how ROC curves are calculated: if those are calculated with fine granularity you essentially get a sensitivity and specificity for "every" threshold value. So, what you could do is simply calculate the sensitivities, specificities and corresponding threshold as if you would want to obtain a ROC curve...

library(pROC)
myRoc <- roc(predictor = predvalues, response = mtcars$vs)
plot(myRoc)
myRoc$specificities
print(with(myRoc, data.frame(specificities, sensitivities, thresholds)))

# specificities sensitivities  thresholds
# 1     0.00000000    1.00000000        -Inf
# 2     0.05555556    1.00000000 0.002462809
# 3     0.11111111    1.00000000 0.003577104
# 4     0.16666667    1.00000000 0.004656164
# 5     0.22222222    1.00000000 0.005191974
# 6     0.27777778    1.00000000 0.006171197
# [...]

...and then lookup the corresponding sensitivities and thresholds for whichever specificities you are interested in, e.g. as:

cutoffs <- c(0.99, 0.90, 0.85, 0.80, 0.75)
myData <- with(myRoc, data.frame(specificities, sensitivities, thresholds))
library(plyr)
print(laply(cutoffs, function(cutoff) myData$sensitivities[which.min(abs(myData$specificities-cutoff))]))

# [1] 0.7857143 0.8571429 0.8571429 0.9285714 0.9285714

Thanks, @geekoverdose! Great idea to think about the ROC-curve. It works, this is what I was looking for! — Hed, Aug 04 '16 at 07:36

Calculate cutoff and sensitivity for specific values of specificity?

1 Answers1