Questions tagged [auc]

The area under the ROC curve can be thought of as a single scalar representation of the ROC curve itself. The AUC of a classifier has the property of being equivalent to the probability that the classifier will rank a randomly chosen positive data point higher than a randomly chosen negative data point.

The area under the ROC curve can be thought of as a single scalar representation of the ROC curve itself. Since this value represents part of the area of a 1x1 square, the AUC is a value between 0.0 and 1.0. However, since a classifier should always perform better than random, the realistic domain of the AUC values should be 0.5 to 1.0. The AUC of a classifier has the property of being equivalent to the probability that the classifier will rank a randomly chosen positive data point higher than a randomly chosen negative data point [Fawcett, 2006]. It can be shown that the AUC is related to the Gini coefficient [Hand et al, 2001]. The AUC can be estimated using trapezoidal approximation by considering the interval between consecutive points [Hand et al, 2001]

Fawcett, Tom. 2006. “An Introduction to ROC Analysis.” Pattern Recognition Letters 27 (8) (June): 861–874. doi:10.1016/j.patrec.2005.10.010.

Hand, David J, and Robert J Till. 2001. “A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems.” Machine Learning 45 (2) (January 1): 171–186. doi:10.1023/A:1010920819831.

535 questions
3
votes
1 answer

Difference between sklearn.roc_auc_score() and sklearn.plot_roc_curve()

I'd like to evaluate my machine learning model. I computed the area under the ROC curve with roc_auc_score() and plotted the ROC curve with plot_roc_curve() functions of sklearn. In the second function the AUC is also computed and shown in the plot.…
hafnerl
  • 123
  • 1
  • 5
3
votes
1 answer

Different results using log_loss/roc_auc_score or performing cross_val_score with scoring='neg_log_loss'/'roc_auc' (Scikit learn)

Exploring some classification models in Scikit learn I noticed that the scores I got for log loss and for ROC AUC were consistently lower while performing cross validation than while fitting and predicting on the whole training set (done to check…
3
votes
1 answer

How this method or formula for calculating ROC AUC works?

I was trying to calculate the AUC using MySQL for the data in table like below: y p 1 0.872637 0 0.130633 0 0.098054 ... ... 1 0.060190 0 0.110938 I came across the following SQL query which is giving the correct AUC score (I verified…
Aditya Jain
  • 129
  • 1
  • 5
3
votes
0 answers

ROC Curve Ranger

I am trying to calculate ROC Curve and AUC using ranger for a binomial classification problem (0 and 1), where the response variable is defined as BiClass. Suppose I cast a data frame to Train_Set and Test_Set (75% and 25 % respectively) and compute…
Ray
  • 321
  • 2
  • 12
3
votes
2 answers

Is it fine to have a threshold greater than 1 in roc_curve metrics?

Predicting the probability of class assignment for each chosen sample from the Train_features: probs = classifier.predict_proba(Train_features)` Choosing the class for which the AUC has to be determined. preds = probs[:,1] Calculating false…
3
votes
5 answers

Why do my models keep getting exactly 0.5 AUC?

I am currently doing a project in which I need to predict eye disease in a group of images. I am using the Keras built-in applications. I am getting good results on VGG16 and VGG19, but on the Xception architecture I keep getting AUC of exactly 0.5…
Janus Syrak
  • 59
  • 1
  • 6
3
votes
1 answer

Individual AUC after multiple imputation using MICE

I have a question about calculating an AUC for every individual in a dataset, after imputation using MICE. I know how I can do it in a complete cases dataset. I have done it as follows: id <- c(1,2,3,4,5,6,7,8,9,10) measure_1 <-…
Anna_70
  • 91
  • 1
  • 7
3
votes
1 answer

scoring "roc_auc" value is not working with gridsearchCV appling RandomForestclassifer

I keep getting this error when perform this with gridsearchCV with scoring value is 'roc_auc'('f1', 'precision','recall' work fine) # Construct a pipeline pipe =…
3
votes
0 answers

Keras callback for model class API

I'd like to compute AUROC (Area Under ROC Curve) for every epoch and store them for later display. My model is Model API. Is it possible to create a similar snippet as follow, which uses callbacks in case of Sequential API? Knowing that self.model,…
snailbee
  • 31
  • 1
3
votes
1 answer

compare the auc of four ROC curves in r

I want to compare the auc of four roc curves in R. I tried roc.test , but this function can just compare two curves roc.test(roc1,roc2) Does R have a function to compare four curves? I know in stata I can compare 4 curves by roccomp. Does any one…
mandy
  • 483
  • 9
  • 20
3
votes
1 answer

How to calculate randomForest training AUC in R

I am sorry for posting this question again but I really need help on this now. I am trying to calculate the AUC of training set of randomForest model in R and there are two ways to calculate this but give different results. The following is a…
annadai
  • 35
  • 1
  • 3
3
votes
1 answer

How to calculate the AUC value for a ranger RF model?

How can I calculate the AUC value for a ranger model ? Ranger is a fast implementation of randomForest algorithm in R. I'm using the following code to build the ranger model for classification purposes, and get the prediction from the model: #Build…
user2947767
  • 1,281
  • 2
  • 9
  • 9
3
votes
0 answers

Calculating AUC for a glm model on data imputed using the MICE package

On a regular dataset, the calls will go like this: library(ROCR) mymodelFit1 <- glm(data = myData, Outcome~ predictor1+ predictor2+ predictor3, family = binomial(link = "logit"))) prob <- predict(mymodelFit1, newdata=myData,…
Wael Hussein
  • 135
  • 1
  • 12
3
votes
0 answers

Scikit learn GridSearchCV AUC performance

I'm using GridSearchCV to identify the best set of parameters for a random forest classifier. PARAMS = { 'max_depth': [8,None], 'n_estimators': [500,1000] } rf = RandomForestClassifier() clf = grid_search.GridSearchCV(estimator=rf,…
Titus Pullo
  • 3,751
  • 15
  • 45
  • 65
3
votes
1 answer

How can I tell h2o deep learning grid to have AUC instead of residual deviance

I would like to measure models performance by looking for AUC or Accuracy. In the grid search I get results with residual deviance,how can I tell h2o deep learning grid to have AUC instead of residual deviance and present the results as atable like…
mql4beginner
  • 2,193
  • 5
  • 34
  • 73