0

I am doing a binary classification in Keras, using DenseNet.

Created weighted classes:

# Assign weights
weight_for_0 = num_normal/(num_normal + num_covid)
weight_for_1 = num_covid/(num_normal + num_covid)
class_weight = {0: weight_for_0, 1: weight_for_1}

# Print
print(f"Weight for class 0: {weight_for_0:.2f}")
print(f"Weight for class 1: {weight_for_1:.2f}")

As a result, I have

Weight for class 0: 0.74
Weight for class 1: 0.26

I fitted the model with class_weight

history_dense201_weighted = model_dense_201.fit_generator(train_generator, epochs = 20, 
validation_data = valid_generator, class_weight = class_weight, callbacks = [# mcp_save,                                                                                                                                                            
early_stopping, tensorboard_callback])

But when I want to evaluate the model, I am not sure how to evaluate the weighted model, because the class_weight is a part of the history.

How to update this code, using instead of default model_dense_201 model a weighted model?

# Evaluation 
evaluation = model_dense_201.evaluate(valid_generator)
print(f"Validation Accuracy: {evaluation[1] * 100:.2f}%")
evaluation = model_dense_201.evaluate(train_generator)
print(f"Train Accuracy: {evaluation[1] * 100:.2f}%")
Anakin Skywalker
  • 2,400
  • 5
  • 35
  • 63

1 Answers1

0

Found this.

https://github.com/tensorflow/tensorflow/issues/35825

Quote from Francois (aka Chollet):

"The reason we don't support class weights in evaluate is that the class_weight argument represents sample weights that are computed from the labels, but the labels should not be an input to the model during evaluation. During training this is fine, but during evaluation this represents a data leak from the labels to your metrics. If you used class weighting in evaluate, your score would not be reproducible on real test data (when you don't have the labels).

So this is conceptually wrong."

Anakin Skywalker
  • 2,400
  • 5
  • 35
  • 63
  • 1
    This may be fine when the reason for weighting is to deal with imbalanced training data. I have an application where it is simply more important to get a certain class right than the others, and I want to see my weighting factor in evaluation. Creating sample weights from class weights is just extra work. – Mastiff Sep 24 '21 at 16:27