8

I was trying to find the best features that dominate for the output of my regression model, Following is my code.

seed = 7
np.random.seed(seed)
estimators = []
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=3,
                           batch_size=20)))
pipeline = Pipeline(estimators)
rfe = RFE(estimator= pipeline, n_features_to_select=5)
fit = rfe.fit(X_set, Y_set)

But I get the following runtime error when running.

RuntimeError: The classifier does not expose "coef_" or "feature_importances_" attributes

How to overcome this issue and select best features for my model? If not, Can I use algorithms like LogisticRegression() provided and supported by RFE in Scikit to achieve the task of finding best features for my dataset?

Klaus
  • 1,641
  • 1
  • 10
  • 22

2 Answers2

17

I assume your Keras model is some kind of a neural network. And with NN in general it is kind of hard to see which input features are relevant and which are not. The reason for this is that each input feature has multiple coefficients that are linked to it - each corresponding to one node of the first hidden layer. Adding additional hidden layers makes it even more complicated to determine how big of an impact the input feature has on the final prediction.

On the other hand, for linear models it is very straightforward since each feature x_i has a corresponding weight/coefficient w_i and its magnitude directly determines how big of an impact it has in prediction (assuming that features are scaled of course).

The RFE estimator (Recursive feature elimination) assumes that your prediction model has an attribute coef_ (linear models) or feature_importances_(tree models) that has the length of input features and that it represents their relevance (in absolute terms).

My suggestion:

  1. Feature selection: (Option a) Run the RFE on any linear / tree model to reduce the number of features to some desired number n_features_to_select. (Option b) Use regularized linear models like lasso / elastic net that enforce sparsity. The problem here is that you cannot directly set the actual number of selected features. (Option c) Use any other feature selection technique from here.
  2. Neural Network: Use only features from (1) for your neural network.
Jan K
  • 4,040
  • 1
  • 15
  • 16
  • Thanks for your answer, to get more clarified, Is it ok if I run RFE with an algorithm like LogisticRegression() provided by Sci-kit to determine the feature importance? and then use my deep learning model with pre determined features obtained with RFE. Also Do I need to feature scale before applying RFE? – Klaus May 09 '18 at 04:11
2

Suggestion:

Perform the RFE algorithm on a sklearn-based algorithm to observe feature importance. Finally, you use the most importantly observed features to train your algorithm based on Keras.

To your question: Standardization is not required for logistic regression

Hendouz
  • 491
  • 1
  • 7
  • 17