Why do the following methods of computing the accuracy ratio give different results?
Approach 1: Cumulative Accuracy Profile (CAP) curve
The accuracy ratio is computed from definition as the difference between the area under curve of the CAP of the trained model and that of a random model, divided by the difference between the area under curve of the CAP of a perfect model and that of a random model.
Approach 2: Receiver Operating Characteristic (ROC) curve. We compute the area under the ROC curve, and use the statistic
AR = Gini = 2*(Area Under ROC Curve) - 1
For a derivation of the statistics, please refer to the paper "Measuring the Discriminative Power of Rating Systems" (https://www.bundesbank.de/resource/blob/704150/b9fa10a16dfff3c98842581253f6d141/mL/2003-10-01-dkp-01-data.pdf)
For the source of the code I'm using and more examples, please refer to the article "Machine Learning Classifier Evaluation using ROC" (https://towardsdatascience.com/machine-learning-classifier-evaluation-using-roc-and-cap-curves-7db60fe6b716)
In what follows, we have a vector y_test of actual test labels, with a vector test_pred_probs of predicted probabilities of the positive class (label=1) that is output from some predictive model. We compute the accuracy ratio of this prediction.
from sklearn.metrics import roc_curve, auc, roc_auc_score
import numpy as np
y_test = [0,1,1,0,1,1,1,0,0,1]
test_pred_probs = [0.2,0.4,0.1,0,0,1,0.9,0.3,0.2,0.8]
total = len(y_test)
one_count = np.sum(y_test)
zero_count = total - one_count
lm = [y for _,y in sorted(zip(test_pred_probs,y_test),reverse=True)]
x = np.arange(0,total+1)
y = np.append([0],np.cumsum(lm))
a = auc([0,total],[0,one_count])
aP = auc([0,one_count,total],[0,one_count,one_count]) - a
aR = auc(x,y) - a
print("Acc ratio:",aR/aP) #returns 0.5
print("Acc ratio from ROC curve:",2*roc_auc_score(y_test,test_pred_probs)-1) #returns 0.458
In some cases, these two approaches give drastically different results - the former gives an AR of about 0.7, and the latter using ROC method gives an AR of 0.12.
