Pass estimator to custom score function via sklearn.metrics.make_scorer

Question

I'd like to make a custom scoring function involving classification probabilities as follows:

def custom_score(y_true, y_pred_proba):
    error = ...
    return error

my_scorer = make_scorer(custom_score, needs_proba=True)

gs = GridSearchCV(estimator=KNeighborsClassifier(),
                  param_grid=[{'n_neighbors': [6]}],
                  cv=5,
                  scoring=my_scorer)

Is there any way to pass the estimator, as fit by GridSearch with the given data and parameters, to my custom scoring function? Then I could interpret the probabilities using estimator.classes_

For example:

def custom_score(y_true, y_pred_proba, clf):
    class_labels = clf.classes_
    error = ...
    return error

score 14 · Answer 1 · answered Jun 27 '16 at 23:42

14

There is an alternative way to make a scorer mentioned in the documentation. Using this method I can do the following:

def my_scorer(clf, X, y_true):
    class_labels = clf.classes_
    y_pred_proba = clf.predict_proba(X)
    error = ...
    return error

gs = GridSearchCV(estimator=KNeighborsClassifier(),
                  param_grid=[{'n_neighbors': [6]}],
                  cv=5,
                  scoring=my_scorer)

This avoids the use of sklearn.metrics.make_scorer.

answered Jun 27 '16 at 23:42

Alex

12,078
6
64
74

You can not use `classes_` attribute of `clf` in my_scorer because at the time of the execution of `my_scorer` function, the classifier isn't fit yet. – quents Jul 09 '18 at 10:43
Thanks! The only problem is when you have multiple scorers you run predict_proba once per scorer – jcp Dec 06 '18 at 16:19
How does gridsearch use the best params for my_scorer function like lower the error is better or higher – Ankit Kumar Namdeo Apr 29 '21 at 10:19

score 4 · Answer 2 · answered Oct 20 '17 at 07:15

According to make_scorer docs, it receives **kwargs : additional arguments as additional parameters to be passed to score_func.

So you can just write your score function as:

def custom_score(y_true, y_pred_proba, clf):
    class_labels = clf.classes_
    error = ...
    return error

Then use make_scorer as:

my_scorer = make_scorer(custom_score, needs_proba=True, clf=clf_you_want)

The benefit of this method is you can pass any other param to your score function easily.

Pass estimator to custom score function via sklearn.metrics.make_scorer

2 Answers2

Linked