1

I'm trying to use exact match / subset accuracy as a metric for my Keras model. I understand basically how it's supposed to work, but I'm having a hard time with the tensor manipulation.

I'm working on a multilabel classification task with 55 possible labels. I'm considering an output > 0.5 to be a positive for that label. I want a metric that describes how often the output exactly matches the true labels.

My approach is to convert y_true to tf.bool, and y_pred > 0.5 to tf.bool, and then return a tensor containing True if they match exactly, and False otherwise. It appears to be working when I do basic tests, but when I train the model, it stays at 0.0000 without ever changing.

def subset_accuracy(y_true, y_pred):
    y_pred_bin = tf.cast(y_pred > 0.5, tf.bool)
    equality = tf.equal(tf.cast(y_true, tf.bool), y_pred_bin)
    return tf.equal(
        tf.cast(tf.math.count_nonzero(equality), tf.int32),
        tf.size(y_true)
    )

I am expecting to see the metric slowly climb, even if it only goes up to 50% or something. But it's staying at 0.0.

Dawson Eliasen
  • 15
  • 1
  • 5

2 Answers2

0

Suggestion: test your metric independently from the model. Use a model (untrained works as well) and model.evaluate a single batch. Compute the metric manually by using the output of model.predict.

Make sure that your computation and the metric the model outputs come to the same result and that the result makes sense for the values in this batch.

Once you are sure that your loss is really mathematically correct; you can then try to debug your model.

It is not clear from your code snippet what you consider to be subset accuracy.

For instance Keras defines categorical_acuracy as:

def categorical_accuracy(y_true, y_pred):
    return K.cast(K.equal(K.argmax(y_true, axis=-1),
                          K.argmax(y_pred, axis=-1)),
                  K.floatx())

How do you intend your accuracy metric to be different. Just ensure that the value is greater than 0.5 ? Perhaps you may consider modifying the Keras metric.

Pedro Marques
  • 2,642
  • 1
  • 10
  • 10
  • Thanks for your response. I'll give the manual verification a shot. As far as the `categorical_accuracy` goes, that is not what I'm looking for because it does not really work for instances where there is more than one true label. It only compares to the maximum value of the prediction. I'll try modifying this function to see if it gives what I want. Thanks again. – Dawson Eliasen Jun 24 '19 at 16:17
  • Update: I verified that the function is working as I expect it to based on the output of `model.predict()`. So I'm not sure why it stays at 0.0 during training. – Dawson Eliasen Jun 24 '19 at 17:02
  • categorical_accuracy is designed for problems where there are multiple classes being predicted. It would be typically used when the output layer has a softmax activation and is trained with a categorical cross entropy loss. – Pedro Marques Jun 24 '19 at 19:27
0

Here is another option, tested with tensorflow 2.3:

def subset_accuracy(y_true, y_pred):
    threshold = tf.constant(.8, tf.float32)
    gtt_pred = tf.math.greater(y_pred, threshold)
    gtt_true = tf.math.greater(y_true, threshold)
    accuracy = tf.reduce_mean(tf.cast(tf.equal(gtt_pred, gtt_true), tf.float32), axis=-1)
    return accuracy

I would imagine that tf.cast(y_true, tf.bool) could be a problem, as it casts float to bool, so depending on how tf deals with it internally, it may first get cast to int, so anything < 1.0 will be zero, and then to bool. That's why nothing will match and you only get zero accuracy.

The above suggestion avoids that problem.

r-or
  • 46
  • 1