calculate accuracy and precision of confusion matrix

Question

Is there any tool / R package available to calculate accuracy and precision of a confusion matrix?

The formula and data structure are here.

Answer 1

yes, you can calculate Accuracy and precision in R with confusion matrix. It uses Caret package.

Here is the example :

lvs <- c("normal", "abnormal")
truth <- factor(rep(lvs, times = c(86, 258)),
                levels = rev(lvs))
pred <- factor(
               c(
                 rep(lvs, times = c(54, 32)),
                 rep(lvs, times = c(27, 231))),               
               levels = rev(lvs))

xtab <- table(pred, truth)
# load Caret package for computing Confusion matrix
library(caret) 
confusionMatrix(xtab)

And Confusion Matrix for xtab would be like this :

Confusion Matrix and Statistics

          truth
pred       abnormal normal
  abnormal      231     32
  normal         27     54

               Accuracy : 0.8285
                 95% CI : (0.7844, 0.8668)
    No Information Rate : 0.75
    P-Value [Acc > NIR] : 0.0003097

                  Kappa : 0.5336
 Mcnemar's Test P-Value : 0.6025370

            Sensitivity : 0.8953
            Specificity : 0.6279
         Pos Pred Value : 0.8783
         Neg Pred Value : 0.6667
             Prevalence : 0.7500
         Detection Rate : 0.6715
   Detection Prevalence : 0.7645

       'Positive' Class : abnormal

So here is everything, that you want.

Answer 2

@Harsh Trivedi

byClass allows you to pull out the precision and recall from the summary. PPV is precision. Sensitivity is recall. https://en.wikipedia.org/wiki/Precision_and_recall

library(caret)

result <- confusionMatrix(prediction, truth)
precision <- result$byClass['Pos Pred Value']    
recall <- result$byClass['Sensitivity']

I imagine you want to pull out the precision and recall to calculate the f-measure so here it goes.

f_measure <- 2 * ((precision * recall) / (precision + recall))

I also found this handy online calculator for sanity check. http://www.marcovanetti.com/pages/cfmatrix/?noc=2

-bg

Answer 3

In case anybody is having the same issue as I did, the method confusionMatrix() in caret does indeed give sensitivity/specificity. However, if it is fed an object of type train it will run a different method, confusionMatrix.train() which does not have this information.

The solution is to feed the data and reference manually from the train object (i.e. $pred$pred$ and $pred$obs respectively) to the confusionMatrix() method.

Answer 4

In case someone else is looking: thanks to BGA's answer above I got clearer on how to read the confusionMatrix() output and realized you can get the F-measure right out of the result$ByClass output as F1.

 result$byClass
         Sensitivity          Specificity       Pos Pred Value       Neg Pred Value 
           0.9337442            0.8130531            0.8776249            0.8952497 
           Precision               Recall                   F1           Prevalence 
           0.8776249            0.9337442            0.9048152            0.5894641 
      Detection Rate Detection Prevalence    Balanced Accuracy 
           0.5504087            0.6271571            0.8733987 

Calculating f_measure below with same formula as in above comment also gives 0.9048152.

You can also get the Accuracy from results$overall

result$overall
      Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull AccuracyPValue 
  8.841962e-01   7.573509e-01   8.743763e-01   8.935033e-01   5.894641e-01   0.000000e+00 
 McnemarPValue 
  2.745521e-13

Or use Balanced Accuracy from results