I want to get feature importance in Keras neural network by applying:
import eli5
from eli5.sklearn import PermutationImportance
perm = PermutationImportance(model, scoring='accuracy',
random_state=1).fit(X_train,y_train)
eli5.show_weights(perm, feature_names = X_train.columns.tolist())
and see the error:
Classification metrics can't handle a mix of binary and continuous targets
I've unsuccessfully tried to setup to PermutationImportance, scoring all the metrics from the sklearn list (sorted(sklearn.metrics.SCORERS.keys())):
['accuracy', 'adjusted_mutual_info_score', 'adjusted_rand_score', 'average_precision', 'balanced_accuracy', 'brier_score_loss',
'completeness_score', 'explained_variance', 'f1', 'f1_macro', 'f1_micro',
'f1_samples', 'f1_weighted', 'fowlkes_mallows_score', 'homogeneity_score',
'jaccard', 'jaccard_macro', 'jaccard_micro', 'jaccard_samples',
'jaccard_weighted', 'max_error', 'mutual_info_score', 'neg_log_loss',
'neg_mean_absolute_error', 'neg_mean_squared_error',
'neg_mean_squared_log_error', 'neg_median_absolute_error',
'normalized_mutual_info_score', 'precision', 'precision_macro',
'precision_micro', 'precision_samples', 'precision_weighted', 'r2', 'recall',
'recall_macro', 'recall_micro', 'recall_samples', 'recall_weighted', 'roc_auc',
'v_measure_score']
On the forums I read: this can be caused by the conflict of the purpose (binary classification) and the method (linear regression). But I don't use in linear regression in network (see code below).
Would you advise, please, how to solve the problem. Thank you.
model = Sequential()
model.add(Dense(1200, input_dim=len(X_train.columns), activation='relu'))
model.add(Dense(150, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
loss="binary_crossentropy"
optimizer=adam(lr=0.01)
metrics=['accuracy']
epochs = 2
batch_size = 32
verbose = 0
model.compile(loss=loss,
optimizer=optimizer,
metrics=metrics)
model.fit(X_train, y_train,
epochs = epochs,
batch_size=batch_size,
verbose = verbose)
# feature importance
import eli5
from eli5.sklearn import PermutationImportance
perm = PermutationImportance(model, scoring='accuracy', random_state=1).fit(X_train,y_train)
eli5.show_weights(perm, feature_names = X_train.columns.tolist())