8

I am trying to use GridSearchCV to optimize an analysis I am doing, and I have read that it supports multiple scoring methods, and I have found an example of this method elsewhere (example), but when I attempt to run a GridSearchCV with multiple scoring metrics in multiple formats that are supposed to be supported, it throws an error:

File "/home/graduate/adsherma/miniconda2/envs/testenv/lib/python2.7/site-packages/sklearn/model_selection/_validation.py", line 288, in _score
  score = scorer(estimator, X_test, y_test)
TypeError: 'dict' object is not callable

My source code for this is:

DF = pd.read_pickle("OutPut/from_ntuples/nominal.pkl")
X = DF[RC.FittableParameters]
y = DF['isSignal']

pipe = Pipeline([
    ('reduce_dim', SelectKBest()),
    ('classify', AdaBoostClassifier())
])

BASE_ESTIMATORS = [DecisionTreeClassifier(max_depth=i) for i in range(1, 4)]
N_ESTIMATORS = range(100, 600, 100)

param_grid = [
    {
        'reduce_dim': [PCA()],
        'reduce_dim__n_components': [1,10,20,30,40,50],
        'classify__base_estimator': BASE_ESTIMATORS, 
        'classify__n_estimators': N_ESTIMATORS, 
    } ,
]
scoring = {'Precision': make_scorer(precision_score), 
    'Accuracy': make_scorer(accuracy_score)} #does not work
# scoring = ['accuracy', 'precision'] #Does not work
# scoring = 'accuracy' #works
# scoring = make_scorer(accuracy_score) #works
grid = GridSearchCV(pipe, cv=5, n_jobs=1, 
                    param_grid=param_grid, verbose=1,
                    scoring=scoring)
grid.fit(X, y)

The error will be the same if I try a list or tuple, but will complain that lists and tuples are not callable instead. This is basically copy pasted from the example link above (can't re-link because I don't have enough reputation, apparently) so I am a bit at a loss for where to proceed.

Gambit1614
  • 8,547
  • 1
  • 25
  • 51
Alex
  • 143
  • 6
  • 8
    Which version of scikit are you using? This feature has been added just recently, maybe in version 0.19. – Vivek Kumar Aug 21 '17 at 12:26
  • 1
    I am using 0.18.2, so maybe that is it! Checking the 0.18 documentation it doesn't seem to say anything about supporting multiple scorers. I feel pretty dumb but that seems to cover it! Thanks a lot. – Alex Aug 21 '17 at 14:55
  • you should be doing something like this: def specificity(y_true, y_pred): tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel() specificity = tn / (tn + fp) print(specificity) return specificity scoring = {'specificity': make_scorer(specificity)} – Prometheus Sep 14 '18 at 13:18

0 Answers0