2

I want to plot macro average AUC with three classes. My work example is:

roc <- multiclass.roc(Claims$CLAIMS2 ~ Claims$PredA)
roc

#Call:
#multiclass.roc.formula(formula = Claims$CLAIMS2 ~ Claims$PredA)
#Data: ClaimsA$PredA with 3 levels of ClaimsA$CLAIMS2: >=2, 0, 1.
#Multi-class area under the curve: 0.6905

I found my macro average AUC for three classes of claims is 0.6905. Then, I plotted three classes together in one plot:

rsA <- roc[['rocs']]

par(pty="s")
plot(rsA[[1]],cex.axis=1.8,lwd = 4,grid = FALSE, xlab="",ylab="",legacy.axes = F,colorize=FALSE, col="gray80", print.auc=F,print.auc.x=0.8,print.auc.y=0.33) # plot ROC curve
plot(rsA[[2]],cex.axis=1.8,cex.lab=2,lwd = 4,colorize=FALSE, xlab="",ylab="",grid = FALSE, legacy.axes = F,col="gray50", add=TRUE, print.auc=F,print.auc.x=0.8, print.auc.y=0.33)
plot(rsA[[3]],cex.axis=1.8,cex.lab=2,lwd = 4,colorize=FALSE, xlab="",ylab="",grid = FALSE, legacy.axes = F,col="black", add=TRUE, print.auc=F,print.auc.x=0.8, print.auc.y=0.33)
title(xlab = "1-Specificity", line = 4,cex.lab=2.5)            # Add x-axis text
title(ylab = "Sensitivity", line = 5,cex.lab=2.5)            # Add y-axis text
legend(0.7, 0.35, legend=c(TeX("Claims 0 vs 1: 0.664"), TeX("Claims 0 vs >=2: 0.770"), TeX("Claims 1 vs >=2: 0.637")),
       col=c("black", "gray50", "gray80","red"), lty=c(1, 1, 1, 3),lwd = 4, cex=1.7,
       box.lty=0)

enter image description here

I want to include macro average AUC in the above plot. Is there any way to plot macro average AUC with three classes in one plot?

Quinten
  • 35,235
  • 5
  • 20
  • 53
F74
  • 31
  • 2

2 Answers2

1

My understanding is roc_auc is a single score, computed using the roc_curve. So it doesn't make sense to try to plot it.

Regardless, I was playing around with tidymodels and experimenting with code from the tidymodels book. Here's some minimal code to compute macro-weighted roc_auc, and plot out the underlying roc:

library(tidymodels)
library(palmerpenguins)
#> Attaching package: 'palmerpenguins'
#> The following object is masked from 'package:modeldata':
#> 
#>     penguins

penguins_split <- initial_split(penguins,
                                strata = "species")
penguins_train <- training(penguins_split)

rf_spec <- rand_forest() |>
  set_mode("classification")

results <- workflow(preprocessor = recipe(species ~ island + year,
                                          data = penguins_train),
                    spec = rf_spec) |>
  last_fit(penguins_split)

results |>
  collect_predictions() |>
  roc_auc(
    truth = species ,
    .pred_Adelie,
    .pred_Chinstrap,
    .pred_Gentoo,
    estimator = "macro_weighted"
  )
#> # A tibble: 1 × 3
#>   .metric .estimator     .estimate
#>   <chr>   <chr>              <dbl>
#> 1 roc_auc macro_weighted     0.806

results |>
  collect_predictions() |>
  roc_curve(truth = species ,
            .pred_Adelie, .pred_Chinstrap, .pred_Gentoo) |>
  autoplot()

Created on 2022-07-29 by the reprex package (v2.0.1)

Desmond
  • 1,047
  • 7
  • 14
  • Thanks. My concerns is to plot average auc (roc_auc macro_weighted = 0.806). Is this possible in R? – F74 Jul 31 '22 at 11:59
  • As mentioned in my answer, you can only plot roc_curves. roc_auc is a single score and plotting it entails plotting a single point. – Desmond Aug 01 '22 at 02:29
  • @Desmond For the roc_auc and roc_curve functions, how do you know which order to inpur the .pred_"name" arguments? I find that changing the order (example: .pred_Adelie, .pred_Chinstrap, .pred_Gentoo is different than .pred_Chinstrap, .pred_Gentoo, .pred_Adelie) changes the results. – STEMQs Dec 19 '22 at 21:16
0

I assume you want to add the existing value of the AUC on the plot. As @Desmond remarked, the average is not a curve, and therefore cannot be plotted in itself.

You can easily add the value to your legend:

# Get the AUC into a string
avg_auc <- roc$auc
avg_auc_legend <- sprintf("Average: %.3f", avg_auc)

# Plot the legend text
legend(0.7, 0.35, 
       legend=c("Claims 0 vs 1: 0.664",
                "Claims 0 vs >=2: 0.770", 
                "Claims 1 vs >=2: 0.637", 
                avg_auc_legend), 
       col=c("black", "gray50", "gray80"), 
       lty=c(1, 1, 1, 0), 
       lwd = 4, 
       cex=1.7,
       box.lty=0)
Calimo
  • 7,510
  • 4
  • 39
  • 61
  • Thanks. Yes I already did same. I found macro AUC can be plotted only using Python. Please see link: https://www.scikit-yb.org/en/latest/api/classifier/rocauc.html#multi-class-rocauc-curves – F74 Aug 05 '22 at 09:55