Adjust threshold in cros_val_score sklearn for Lasso

Question

I saw this post that trained a LGBM model but i would like to know how to adapt it for Lasso. I know the prediction may not necessarily be between 0 and 1, but I would like to try this model. I have tried this but it doesnt work:

import numpy as np 
from lightgbm import LGBMClassifier
from sklearn.datasets import make_classification
from sklearn.linear_model import Lasso

X, y = make_classification(n_features=10, random_state=0, n_classes=2, n_samples=1000, n_informative=8)

class Lasso(Lasso):
    def predict(self,X, threshold=0.5):
        result = super(Lasso, self).predict_proba(X)
        predictions = [1 if p>threshold else 0 for p in result[:,0]]
        return predictions

clf = Lasso(alpha=0.05)
clf.fit(X,y)

precision = cross_val_score(Lasso(),X,y,cv=5,scoring='precision')

I get

UserWarning: Scoring failed. The score on this train-test partition for these parameters will be set to nan

AlexK · Answer 1 · 2022-07-09T05:46:11.473

The specific model class you chose (Lasso()) is actually used for regression problems, as it minimizes penalized square loss, which is not appropriate in your case. Instead, you can use LogisticRegression() with L1 penalty to optimize a logistic function with a Lasso penalty. To control the regularization strength, the C= parameter is used (see docs).

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

X, y = make_classification(
    n_features=10, random_state=0, n_classes=2, n_samples=1000, n_informative=8
)

class LassoLR(LogisticRegression):
    def predict(self,X, threshold=0.5):
        result = super(LassoLR, self).predict_proba(X)
        predictions = [1 if p>threshold else 0 for p in result[:,0]]
        return predictions

clf = LassoLR(penalty='l1', solver='liblinear', C=1.)
clf.fit(X,y)

precision = cross_val_score(LassoLR(),X,y,cv=5,scoring='precision')
print(precision)
# array([0.04166667, 0.08163265, 0.1010101 , 0.125     , 0.05940594])

Adjust threshold in cros_val_score sklearn for Lasso

1 Answers1