Questions tagged [auc]

The area under the ROC curve can be thought of as a single scalar representation of the ROC curve itself. The AUC of a classifier has the property of being equivalent to the probability that the classifier will rank a randomly chosen positive data point higher than a randomly chosen negative data point.

The area under the ROC curve can be thought of as a single scalar representation of the ROC curve itself. Since this value represents part of the area of a 1x1 square, the AUC is a value between 0.0 and 1.0. However, since a classifier should always perform better than random, the realistic domain of the AUC values should be 0.5 to 1.0. The AUC of a classifier has the property of being equivalent to the probability that the classifier will rank a randomly chosen positive data point higher than a randomly chosen negative data point [Fawcett, 2006]. It can be shown that the AUC is related to the Gini coefficient [Hand et al, 2001]. The AUC can be estimated using trapezoidal approximation by considering the interval between consecutive points [Hand et al, 2001]

Fawcett, Tom. 2006. “An Introduction to ROC Analysis.” Pattern Recognition Letters 27 (8) (June): 861–874. doi:10.1016/j.patrec.2005.10.010.

Hand, David J, and Robert J Till. 2001. “A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems.” Machine Learning 45 (2) (January 1): 171–186. doi:10.1023/A:1010920819831.

535 questions
5
votes
0 answers

How to pass multiple custom metrics (eval_metric) in python xgboost?

The folloiwng code is not working, where aucerr and aoeerr are custom evaluation metrics, it is working with just one eval_metric either aucerr or aoeerr prtXGB.fit(trainData, targetVar, early_stopping_rounds=10, eval_metric= [aucerr, aoeerr],…
BigDataScientist
  • 1,045
  • 5
  • 17
  • 37
5
votes
2 answers

xgboost using the auc metric correctly

I have a slightly imbalanced dataset for a binary classification problem, with a positive to negative ratio of 0.6. I recently learned about the auc metric from this answer: https://stats.stackexchange.com/a/132832/128229, and decided to use it. But…
Vikash Balasubramanian
  • 2,921
  • 3
  • 33
  • 74
5
votes
3 answers

AUC of a Precision Recall Curve by using package ROCR

How to obtain AUC (area under curve) of a Precision Recall Curve by using package ROCR..? library(ROCR) data(ROCR.simple) pred <- prediction( ROCR.simple$predictions, ROCR.simple$labels) perf <- performance(pred,"tpr","fpr") plot(perf) ##…
Newbie
  • 411
  • 5
  • 18
5
votes
1 answer

Evaluation Metrics for Binary Classification in Spark: AUC and PR curve

I was trying to calculate Precision, Recall by Threshold for LogisticRegressionwithLBFGS using BinaryclassificationMetrics. I got all those. I was trying to figure out if I could get a graphical output of PR and AUC curve. Pasting my Codes…
PARTHA TALUKDER
  • 321
  • 5
  • 17
5
votes
1 answer

R: How to compute AUC and ROC curve for ´bgeva´ objekt/model?

Since I have data with binary response, but rare events, I would like to improve its forecast by fitting a bgeva model instead of a gam model. To prove and compare it´s prediction accuracy and compare it to other models that I tried, I need to…
Peky84
  • 103
  • 7
5
votes
1 answer

not enough distinct predictions to compute area under roc

I'm tying to calculate the AUC using auc(roc(predictions, labels)), where labels is a numeric vector of 1 (x15) and 0 (x500), and predictions is a numeric vector with probabilities derived from a glm [binomial]. It should be very simple, but…
user2942623
  • 559
  • 5
  • 8
5
votes
1 answer

Python: integrating area under curve with uneven steps in x

I have a list of y values and a list of x values. I would like to find the area under the curve defined by these points. I have found a couple of solutions to this problem for x values with even spacing: 1) Calculating the area under a curve given…
user3390452
  • 51
  • 1
  • 2
4
votes
1 answer

avoid x is neither increasing nor decreasing : {}.".format(x) when calculate auc score

So i have a list of false positive rates and a list of true positive rates obtained by varying a certain threshold. I'm trying to calculate the auc score but unfortunately i can't use roc_auc_score method from scikit-learn so i'm using the more…
JayJona
  • 469
  • 1
  • 16
  • 41
4
votes
2 answers

How does the predict function of StatsModels interact with roc_auc_score of scikit-learn?

I am trying to understand the predict function in Python statsmodels for a Logit model. Its documentation is here. When I build a Logit Model and use predict, it returns values from 0 to 1 as opposed to 0 or 1. Now I read this saying these are…
4
votes
2 answers

Plotting ROC curve for RidgeClassifier in Python

I want to plot the ROC curve for the RidgeClassifier. But the code comes with an error: I googled for solutions and it comes up to change predict_proba to predict, but it does not work! predY =…
Flodude
  • 255
  • 2
  • 12
4
votes
0 answers

Appropriate use of ROC and AUC in keras model for multi-class, multi-label tasks

I am trying to understand the appropriate use of AUC ROC in my keras model. The specific model I am training is a multi-class, multi-lab el classifier, where each sample can belong to multiple classes. the code is as follows (starting at one-hot…
user9317212
4
votes
1 answer

Tensorflow 2 Metrics produce wrong results with 2 GPUs

I took this piece of code from tensorflow documentation about distributed training with custom loop https://www.tensorflow.org/tutorials/distribute/custom_training and I just fixed it to work with the tf.keras.metrics.AUC and run it with 2 GPUS (2…
Ζι Βάγγο
  • 184
  • 1
  • 9
4
votes
0 answers

The gof function from package btergm gives AUC value of a precision-recall greater than 1

I was trying to do out-of-sample prediction using the gof function from package btergm. When calculating the AUC value of a precision-recall curve from the testing set, I get the result of 1.012909, which seems to be theoretically impossible. How…
4
votes
1 answer

ROC curve says my predictions are worse than random but my confusion matrix says otherwise

I am working with data to classify handwritten numbers from 0 to 9. I am using PCA to reduce the dimensionality to 6 principal components and KNN to model the data. When I created the confusion matrix, I got reasonable answers out. It wasn't…
4
votes
1 answer

How do I create multiple custom AUC metrics, one for each of the outputs, in TensorFlow?

In TensorFlow 2.0, there's the class tf.keras.metrics.AUC. It can easily be added to the list of metrics of the compile method as follows. # Example taken from the documentation model.compile('sgd', loss='mse',…
nbro
  • 15,395
  • 32
  • 113
  • 196
1 2
3
35 36