14

I want to score different classifiers with different parameters.

For speedup on LogisticRegression I use LogisticRegressionCV (which at least 2x faster) and plan use GridSearchCV for others.

But problem while it give me equal C parameters, but not the AUC ROC scoring.

I'll try fix many parameters like scorer, random_state, solver, max_iter, tol... Please look at example (real data have no mater):

Test data and common part:

from sklearn import datasets
boston = datasets.load_boston()
X = boston.data
y = boston.target
y[y <= y.mean()] = 0; y[y > 0] = 1

import numpy as np
from sklearn.cross_validation import KFold
from sklearn.linear_model import LogisticRegression
from sklearn.grid_search import GridSearchCV
from sklearn.linear_model import LogisticRegressionCV

fold = KFold(len(y), n_folds=5, shuffle=True, random_state=777)

GridSearchCV

grid = {
    'C': np.power(10.0, np.arange(-10, 10))
     , 'solver': ['newton-cg']
}
clf = LogisticRegression(penalty='l2', random_state=777, max_iter=10000, tol=10)
gs = GridSearchCV(clf, grid, scoring='roc_auc', cv=fold)
gs.fit(X, y)

print ('gs.best_score_:', gs.best_score_)

gs.best_score_: 0.939162082194

LogisticRegressionCV

searchCV = LogisticRegressionCV(
    Cs=list(np.power(10.0, np.arange(-10, 10)))
    ,penalty='l2'
    ,scoring='roc_auc'
    ,cv=fold
    ,random_state=777
    ,max_iter=10000
    ,fit_intercept=True
    ,solver='newton-cg'
    ,tol=10
)
searchCV.fit(X, y)

print ('Max auc_roc:', searchCV.scores_[1].max())

Max auc_roc: 0.970588235294

Solver newton-cg used just to provide fixed value, other tried too. What I forgot?

P.S. In both cases I also got warning "/usr/lib64/python3.4/site-packages/sklearn/utils/optimize.py:193: UserWarning: Line Search failed warnings.warn('Line Search failed')" which I can't understand too. I'll be happy if someone also describe what it mean, but I hope it is not relevant to my main question.

EDIT UPDATES

By @joeln comment add max_iter=10000 and tol=10 parameters too. It does not change result in any digit, but the warning disappeared.

Hubbitus
  • 5,161
  • 3
  • 41
  • 47
  • 1
    Just to be sure, could you reduce `tol` and provide that same large `max_iter` to `LogisticRegression`? If these are solving the same equation and arriving at a different minimum, it may be because it's stopping the search too early. – joeln Mar 29 '16 at 01:01
  • Thanks. I add max_iter=10000 and tol=10 (unsure what it mean) and correct question. It does not change result in any digit. – Hubbitus Mar 29 '16 at 07:55
  • Curious. I suggested reducing, not increasing `tol`, whose default is 1e-4. In searching (as in Newton's method) around the minimum of the objective function, `tol` defines what small change is close enough to zero to say the minimum has been found. A smaller `tol` ensures optimisations with different initialisations more likely agree. I'm not certain, but I think you should raise this as an issue on the scikit-learn tracker. `LogisticRegressionCV` is relatively new and there may be things to iron out or document better. You may need to provide some data for which this effect occurs. – joeln Mar 29 '16 at 23:52
  • Sorry for the `tol` value. What value are you suggesting? Unfortunately I newbee in ML and also uncertain... What concerned data - my example in question especially based on embedded in sklearn Boston houses demo values and fully reproducible! So, I expect what anyone just copy, paste and run it should receive exact same results. – Hubbitus Mar 30 '16 at 07:24
  • And on all `tol` less than 10**-3 aforementioned warning appeared. On tol=(10**-7) auc_roc still 0.939162082194 and 0.970588235294 respectively. – Hubbitus Mar 30 '16 at 08:32
  • It's good to hear this is on a shared dataset. Please raise this issue in the scikit-learn issue tracker. – joeln Mar 31 '16 at 01:12
  • I had submit issue: https://github.com/scikit-learn/scikit-learn/issues/6619 – Hubbitus Apr 03 '16 at 19:20
  • 1
    Haha sorry I didn't notice how you were printing the score from `LogisticRegressionCV`. That seems to be the main source of error. – joeln Apr 04 '16 at 23:57
  • It seams so. But results still differ... – Hubbitus Apr 05 '16 at 00:03
  • No, the difference is within `tol` – joeln Apr 05 '16 at 00:04
  • @joeln if your wish post answer with that finding summarizing bug and comment answers - I'll accept it. – Hubbitus Jun 01 '16 at 08:37

1 Answers1

10

Here is a copy of the answer by Tom on the scikit-learn issue tracker:

LogisticRegressionCV.scores_ gives the score for all the folds. GridSearchCV.best_score_ gives the best mean score over all the folds.

To get the same result, you need to change your code:

print('Max auc_roc:', searchCV.scores_[1].max())  # is wrong
print('Max auc_roc:', searchCV.scores_[1].mean(axis=0).max())  # is correct

By also using the default tol=1e-4 instead of your tol=10, I get:

('gs.best_score_:', 0.939162082193857)
('Max auc_roc:', 0.93915947999923843)

The (small) remaining difference might come from warm starting in LogisticRegressionCV (which is actually what makes it faster than GridSearchCV).

ogrisel
  • 39,309
  • 12
  • 116
  • 125