2

I want to write a custom metric in Keras (python) to evaluate the performance of my sequence to sequence model as I train. Sequences are one-hot encoded and the tokens are words instead of characters. I want it to report the number of sequences that are FULLY correct only (percentage would be okay too). So if it predicts "The cat sits on the mat" instead of "the cat sat on the mat" that would be incorrect and would not be counted.

Keras has (in my opinion) limited documentation on writing custom metrics (here https://keras.io/metrics/). Basically, what I want is something like

    import keras.backend as K

    def num_correct(y_true, y_pred):
        return K.sum(float(y_pred == y_true))

Now, Keras doesn't like this. It says that float has no attribute shape. I know that y_true and y_pred are tensors, but I'm not fully sure what it wants returned. Help is appreciated.

Hannah Morgan
  • 133
  • 1
  • 9
  • 1
    How is this different from [categorical_accuracy](https://github.com/keras-team/keras/blob/49f5b931410bc2e56378f20a15e8ac919e0efb88/keras/metrics.py#L30)? – nuric Jun 14 '18 at 15:22
  • categorical_accuracy compares the entries token by token instead of sequence by sequence. Thus the above example would not get an accuracy of 0, which is what is desired. – Hannah Morgan Jun 15 '18 at 20:44

1 Answers1

0

First y_pred == y_true checks if the two values and return true if values equal and false otherwise. But I don't think you want a single boolean value to be returned from == operation. If that is the case change the code to

import keras.backend as K

def num_correct(y_true, y_pred):
    equal = K.equal(y_pred,y_true)

    return K.sum(K.cast(equal,K.floatx()))

Secondly even if the returned value from == operation is tensor you can't call float function on tensors. Keras backend have nice method to do this(K.cast).

Mitiku
  • 5,337
  • 3
  • 18
  • 35