How to get coefficients and feature importances from MultiOutputRegressor?

Question

I am trying to perform a MultiOutput Regression using ElasticNet and Random Forests as follows:

from sklearn.ensemble import RandomForestRegressor
from sklearn.multioutput import MultiOutputRegressor
from sklearn.linear_model import ElasticNet

X_train, X_test, y_train, y_test = train_test_split(X_features, y, test_size=0.30,random_state=0)

Elastic Net

l1_range=np.arange(0.1,1.05,0.1).tolist()

regr_Enet=ElasticNetCV(cv=5,copy_X=True,n_alphas=100,l1_ratio=l1_range,selection='cyclic',normalize=False,verbose =2,n_jobs=1)

regr_multi_Enet= MultiOutputRegressor(regr_Enet)##ElasticNetCV

regr_multi_Enet.fit(X_train, y_train)

Random Forest

max_depth = 20
number_of_trees=100

regr_multi_RF=MultiOutputRegressor(RandomForestRegressor(n_estimators=number_of_trees,max_depth=max_depth,random_state=0,n_jobs=1,verbose=1))

regr_multi_RF.fit(X_train, y_train)

y_multirf = regr_multi_RF.predict(X_test)

Everything is going well, however I haven't found a way to obtain the coefficients (coef_ ) or most important features (feature_importances_) of the model. When I write:

regr_multi_Enet.coef_
regr_multi_RF.feature_importances_

It shows the following error:

AttributeError: 'MultiOutputRegressor' object has no attribute 'feature_importances_'
AttributeError: 'MultiOutputRegressor' object has no attribute 'coef_'

I have read the documentation on MultiOutputRegressor but I cannot find a way to extract the coefficients. How to retrieve them?

desertnaut · Accepted Answer · 2018-10-04T14:48:21.470

MultiOutputRegressor itself doesn't have these attributes - you need to access the underlying estimators first using the estimators_ attribute (which, although not mentioned in the docs, it exists indeed - see the docs for MultiOutputClassifier). Here is a reproducible example:

from sklearn.multioutput import MultiOutputRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import ElasticNet

# dummy data
X = np.array([[-1, -1], [-2, -1], [1, 1], [2, 1]])
W = np.array([[1, 1], [1, 1], [2, 2], [2, 2]])

regr_multi_RF=MultiOutputRegressor(RandomForestRegressor())
regr_multi_RF.fit(X,W)

# how many estimators?
len(regr_multi_RF.estimators_)
# 2

regr_multi_RF.estimators_[0].feature_importances_
# array([ 0.4,  0.6])

regr_multi_RF.estimators_[1].feature_importances_
# array([ 0.4,  0.4])

regr_Enet = ElasticNet()
regr_multi_Enet= MultiOutputRegressor(regr_Enet)
regr_multi_Enet.fit(X, W)

regr_multi_Enet.estimators_[0].coef_
# array([ 0.08333333,  0.        ])

regr_multi_Enet.estimators_[1].coef_
# array([ 0.08333333,  0.        ])

score 2 · Answer 2 · answered Oct 04 '18 at 14:19

2

regr_multi_Enet.estimators_[0].coef_

To get the coefficients of the first estimator etc.

answered Oct 04 '18 at 14:19

user2653663

2,818
1
18
22

Thanks for the quick answer! For the sake of completeness: in the case of random forest - regr_multi_RF.estimators_[0].feature_importances_ – Julioolvera Oct 04 '18 at 14:35
doesn't work for knn = KNeighborsRegressor(n_neighbors=3, leaf_size=20) regr = MultiOutputRegressor(knn) – Soyol May 27 '19 at 02:16

Sucheta Swetlana · Answer 3 · 2023-02-20T11:08:46.687

    from sklearn.multioutput import MultiOutputRegressor
        
    from sklearn.ensemble import GradientBoostingRegressor
        
    params = {'n_estimators': 500, 'max_depth':3, 'learning_rate': 0.1} 
        
    reg = MultiOutputRegressor(GradientBoostingRegressor(**params, random_state = 123))
    reg.fit(X_train,Y_train)
    
    ###  feature importance 
    reg.estimators_[0]           # for estimator 0
    reg.estimators_[0].feature_importances_
    print (feature_importance) 

    sorted_idx = np.argsort(feature_importance)
    pos = np.arange(sorted_idx.shape[0]) + 0.5
    fig = plt.figure(figsize=(12, 6))
    plt.subplot(1, 2, 1)
    plt.barh(pos, feature_importance[sorted_idx], align="center")
    plt.yticks(pos, np.array(X.columns)[sorted_idx])
    plt.title("Feature Importance")

While this code snippet may be the solution, including a detailed explanation really helps to improve the quality of your post. Remember that you are answering the question for readers in the future, and those people might not know the reasons for your code suggestion. — Shawn Hemelstrand, Feb 25 '23 at 16:41

How to get coefficients and feature importances from MultiOutputRegressor?

3 Answers3

Linked