Questions tagged [auc]

The area under the ROC curve can be thought of as a single scalar representation of the ROC curve itself. The AUC of a classifier has the property of being equivalent to the probability that the classifier will rank a randomly chosen positive data point higher than a randomly chosen negative data point.

The area under the ROC curve can be thought of as a single scalar representation of the ROC curve itself. Since this value represents part of the area of a 1x1 square, the AUC is a value between 0.0 and 1.0. However, since a classifier should always perform better than random, the realistic domain of the AUC values should be 0.5 to 1.0. The AUC of a classifier has the property of being equivalent to the probability that the classifier will rank a randomly chosen positive data point higher than a randomly chosen negative data point [Fawcett, 2006]. It can be shown that the AUC is related to the Gini coefficient [Hand et al, 2001]. The AUC can be estimated using trapezoidal approximation by considering the interval between consecutive points [Hand et al, 2001]

Fawcett, Tom. 2006. “An Introduction to ROC Analysis.” Pattern Recognition Letters 27 (8) (June): 861–874. doi:10.1016/j.patrec.2005.10.010.

Hand, David J, and Robert J Till. 2001. “A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems.” Machine Learning 45 (2) (January 1): 171–186. doi:10.1023/A:1010920819831.

535 questions
9
votes
2 answers

Plot ROC curve from Cross-Validation (training) data in R

I would like to know if there is a way to plot the average ROC Curve from the cross-validation data of a SVM-RFE model generated with the caret package. My results are: Recursive feature selection Outer resampling method: Cross-Validated (10 fold,…
Rafa OR
  • 339
  • 2
  • 3
  • 8
8
votes
2 answers

Does tf.keras.metrics.AUC work on multi-class problems?

I have a multi-class classification problem and I want to measure AUC on training and test data. tf.keras has implemented AUC metric (tf.keras.metrics.AUC), but I'm not be able to see whether this metric could safely be used in multi-class problems.…
8
votes
1 answer

R get AUC and plot multiple ROC curves together at the same time

I have tried 2 methods to plot ROC curve and get AUC for each ROC curve. Method 1 - The first method is simple but I don't know how to plot multiple ROC curves together. I am simply using roc.curve(hacide.test$cls, pred_rose[,2]), the output will…
Cherry Wu
  • 3,844
  • 9
  • 43
  • 63
8
votes
1 answer

Scikit-learn : roc_auc_score

I am using the roc_auc_score function from scikit-learn to evaluate my model performances. Howver, I get differents values whether I use predict() or predict_proba() p_pred = forest.predict_proba(x_test) y_test_predicted= forest.predict(x_test) fpr,…
user4640449
  • 589
  • 1
  • 5
  • 9
7
votes
2 answers

How to calculate AUC for random forest model in sklearn?

The label in my data is a (N by 1) vector. The label values are either 0 for negative samples or 1 for positive samples (so, it's a binary classification problem). I use the .fit function of sklearn and fitted a random forest on my train set. To…
khemedi
  • 774
  • 3
  • 9
  • 19
7
votes
2 answers

Use TensorFlow loss Global Objectives (recall_at_precision_loss) with Keras (not metrics)

Background I have a multi-label classification problem with 5 labels (e.g. [1 0 1 1 0]). Therefore, I want my model to improve at metrics such as fixed recall, precision-recall AUC or ROC AUC. It doesn't make sense to use a loss function (e.g.…
NumesSanguis
  • 5,832
  • 6
  • 41
  • 76
7
votes
2 answers

How to get ROC curve for decision tree?

I am trying to find ROC curve and AUROC curve for decision tree. My code was something like clf.fit(x,y) y_score = clf.fit(x,y).decision_function(test[col]) pred =…
kanav anand
  • 411
  • 1
  • 4
  • 14
7
votes
1 answer

Computing AUC and ROC curve from multi-class data in scikit-learn (sklearn)?

I am trying to use the scikit-learn module to compute AUC and plot ROC curves for the output of three different classifiers to compare their performance. I am very new to this topic, and I am struggling to understand how the data I have should…
Suriname0
  • 527
  • 1
  • 8
  • 21
7
votes
1 answer

Feature Selection in caret rfe + sum with ROC

I have been trying to apply recursive feature selection using caret package. What I need is that ref uses the AUC as performance measure. After googling for a month I cannot get the process working. Here is the code I have…
José Palma
  • 71
  • 1
  • 3
6
votes
3 answers

sklearn classification metric auc return ValueError

I'm building a two class classification model using KNN I tried to calculate auc_score with from sklearn.metrics import auc auc(y_test, y_pred) --------------------------------------------------------------------------- ValueError …
Jeeth
  • 2,226
  • 5
  • 24
  • 60
6
votes
1 answer

Keras, auc on validation set during training does not match with sklearn auc

I am using my test set as a validation set. I used similar approach as How to compute Receiving Operating Characteristic (ROC) and AUC in keras? The issue is that my val_auc during the training is around 0.85, how ever, when I use fpr, tpr, _ =…
Behinoo
  • 395
  • 5
  • 18
6
votes
2 answers

Manually calculate AUC

How can I obtain the AUC value having fpr and tpr? Fpr and tpr are just 2 floats obtained from these formulas: my_fpr = fp / (fp + tn) my_tpr = tp / (tp + fn) my_roc_auc = auc(my_fpr, my_tpr) I know this can't pe possible, because fpr and tpr are…
user9370613
5
votes
2 answers

Calculating AUC ratio in R

I am generating ecological niche models for a set of species and I would like to use AUC as a metric for ecological niche quality. Steven Phillips, who developed Maxent, provides code in his Maxent manual for calculating the AUC in R. However, I am…
Pascal
  • 1,590
  • 2
  • 16
  • 35
5
votes
1 answer

Which is the correct way to calculate AUC with scikit-learn?

I noticed that the result of the following two codes is different. #1 metrics.plot_roc_curve(classifier, X_test, y_test, ax=plt.gca()) #2 metrics.plot_roc_curve(classifier, X_test, y_test, ax=plt.gca(), label=clsname + ' (AUC = %.2f)' %…
David Ws.
  • 134
  • 2
  • 9
5
votes
1 answer

Proper inputs for Scikit Learn roc_auc_score and ROC Plot

I am trying to determine roc_auc_score for a fit model on a validation set. I am seeing some conflicting information on function inputs. Documentation says: "y_score array-like of shape (n_samples,) or (n_samples, n_classes) Target scores. In the…
Windstorm1981
  • 2,564
  • 7
  • 29
  • 57
1
2
3
35 36