classification report using YellowBrick

Question

i recently implemented Probabilistic Neural Network on the iris dataset. i was trying to print classification report using YellowBrick classifier but when i run this code i get an error. which is as following.

from neupy import algorithms
model = algorithms.PNN(std=0.1, verbose=True, batch_size = 500)
model.train(X_train, Y_train)
predictions = model.predict(X_test)


from yellowbrick.classifier import ClassificationReport
visualizer = ClassificationReport(model, support=True)

visualizer.fit(X_train, Y_train)  # Fit the visualizer and the model
visualizer.score(X_test, Y_test)  # Evaluate the model on the test data
visualizer.show()

this code returns this error.

YellowbrickTypeError: This estimator is not a classifier; try a regression or clustering score visualizer instead!

when i tried the same classification report code for other classification model it works. i have no idea. why is this happening? can anyone help me out with this?

hi @desertnaut.. i have added it. im sorry for not adding it earlier — gendry, Nov 11 '19 at 15:03

score 5 · Answer 1 · answered Nov 11 '19 at 16:28

Yellowbrick is intended to be used with scikit-learn and uses sklearn's type checking system to detect if a model fits a particular class of machine learning problem. If the neupy PNN model implements the scikit-learn estimator API (e.g. fit() and predict()) - it may be possible to directly use the model and bypass the type checking by using the force_model=True parameter as follows:

visualizer = ClassificationReport(model, support=True, force_model=True)

However, after taking a quick glance at the neupy documentation it appears that this won't necessarily work since the neupy method is named train and not fit and because the PNN model does not implement a score() method nor have support for _ suffixed learned parameters.

The solution is to create a light-weight wrapper that surrounds the PNN model that exposes it as a sklearn estimator. Testing on a Yellowbrick dataset, this appears to work:

from sklearn import metrics
from neupy import algorithms
from sklearn.base import BaseEstimator
from yellowbrick.datasets import load_occupancy
from yellowbrick.classifier import ClassificationReport
from sklearn.model_selection import train_test_split


class PNNWrapper(algorithms.PNN, BaseEstimator):
    """
    The PNN wrapper implements BaseEstimator and allows the classification
    report to score the model and understand the learned classes.
    """

    @property
    def classes_(self):
        return self.classes

    def score(self, X_test, y_test):
        y_hat = self.predict(X_test)
        return metrics.accuracy_score(y_test, y_hat)


# Load the binary classification dataset 
X, y = load_occupancy()
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Create and train the PNN model using the sklearn wrapper
model = PNNWrapper(std=0.1, verbose=True, batch_size=500)
model.train(X_train, y_train)

# Create the classification report
viz = ClassificationReport(
    model, 
    support=True, 
    classes=["not occupied", "occupied"], 
    is_fitted=True, 
    force_model=True, 
    title="PNN"
)

# Score the report and show it
viz.score(X_test, y_test)
viz.show()

Although neupy is not currently supported by Yellowbrick, if you're interested - it might be worth submitting an issue suggesting that neupy be added to contrib, similar to how statsmodels is implemented in Yellowbrick.

Can you please guide me on how to use yellowbrick for xgb.train() models? I am fairly new to this module and cannot make it work for my XGB model — Alirezaro93, Aug 09 '21 at 23:25

classification report using YellowBrick

1 Answers1