3

I have a simple ROC plot that I am creating using pROC package:

plot.roc(response, predictor)

It is working fine, as expected, but I would like to add an "ideally" shaped reference curve with AUC 0.8 for comparison (the AUC of my ROC plot is 0.66).

Any thoughts?

Just to clarify, I am not trying to smoothen my ROC plot, but trying to add a reference curve that would represent AUC 0.8 (similar to the reference diagonal line representing AUC 0.5).

David Robinson
  • 77,383
  • 16
  • 167
  • 187
Oposum
  • 1,155
  • 3
  • 22
  • 38

2 Answers2

3

The reference diagonal line has a meaning (a model that guesses randomly), so you would similarly have to define the model associated with your reference curve of AUC 0.8. Different models would be associated with different reference curves.

For instance, one might define a model for which predicted probabilities are distributed evenly between 0 and 1 and for a point with predicted probability p, the probability of the true outcome is p^k for some constant k. It turns that for this model, k=2 yields a plot with AUC 0.8.

library(pROC)
set.seed(144)
probs <- seq(0, 1, length.out=10000)
truth <- runif(10000)^2 < probs
plot.roc(truth, probs)
# Call:
# plot.roc.default(x = truth, predictor = probs)
# 
# Data: probs in 3326 controls (truth FALSE) < 6674 cases (truth TRUE).
# Area under the curve: 0.7977

enter image description here

Some algebra shows that this particular family of models has AUC (2+3k)/(2+4k), meaning it can generate curves with AUC between 0.75 and 1 depending on the value of k.

Another approach you could use is linked to logistic regression. If you had logistic regression linear predictor function value p, aka you would have predicted probability 1/(1+exp(-p)), then you could label the true outcome as true if p plus some normally distributed noise exceeds 0 and otherwise label the true outcome as false. If the normally distributed noise has variance 0 your model will have AUC 1, and if the normally distributed noise has variance approaching infinity your model will have AUC 0.5.

If I assume the original predictions are drawn from the standard normal distribution, it looks like normally distributed noise with standard deviation 1.2 give AUC 0.8 (I couldn't figure out a nice closed form for AUC, though):

set.seed(144)
pred.fxn <- rnorm(10000)
truth <- (pred.fxn + rnorm(10000, 0, 1.2)) >= 0
plot.roc(truth, pred.fxn)
# Call:
# plot.roc.default(x = truth, predictor = pred.fxn)
# 
# Data: pred.fxn in 5025 controls (truth FALSE) < 4975 cases (truth TRUE).
# Area under the curve: 0.7987

enter image description here

josliber
  • 43,891
  • 12
  • 98
  • 133
  • It looks how it should look, except for the fact that it is not 100% smooth for some reason. Otherwise this is what I was talking about. Thanks. – Oposum Apr 18 '15 at 01:01
  • @Oposum the lack of smoothness is likely just that it wasn't constructed with enough samples. You might try 100,000 samples instead of the 10,000 that I have posted here. – josliber Apr 18 '15 at 02:00
2

A quick/rough way is to add a circle of radius 1 onto your plot which will have AUC pi/4 = 0.7853982

library(pROC)
library(car)

n <- 100L

x1 <- rnorm(n, 2.0, 0.5)
x2 <- rnorm(n, -1.0, 2)
y <- rbinom(n, 1L, plogis(-0.4 + 0.5 * x1 + 0.1 * x2))

mod <- glm(y ~ x1 + x2, "binomial")
probs <- predict(mod, type = "response")

plot(roc(y, probs))
ellipse(c(0, 0), matrix(c(1,0,0,1), 2, 2), radius = 1, center.pch = FALSE, col = "blue")

roc

Jeff
  • 718
  • 8
  • 20
  • Roc curves are rarely (if at all) as round as circles. I like how smooth it is, but I was imagining more like this figure http://gim.unmc.edu/dxtests/roccomp.jpg something between the yellow and pink curves. You can certainly tell that none of them are circles (especially the yellow one). – Oposum Apr 18 '15 at 01:04