add custom loss to eval_metric_ops

Question

I have defined my own loss function using sequence_loss

loss = tf.contrib.legacy_seq2seq.sequence_loss(logits, labels, weights)

I am hoping to add this to eval_metric_ops so that in my ML engine package I can display the evaluation loss in tensorboard continuously (defaults are just accuracy).I tried adding this as a custom eval_metric_ops

eval_metric_ops = {
    'loss': loss # this has already been coputed for Modes.EVAL
}

However, I get the error "TypeError: Values of eval_metric_ops must be (metric_value, update_op) tuples, given: Tensor("sequence_loss/truediv:0", shape=(), dtype=float32) for key: loss" -- what do I need to do to pass the loss as an eval_metric_op? I guessing that my current loss should be the metric value, but I'm not sure what the update_op should be?

score 3 · Accepted Answer · answered Aug 16 '17 at 19:18

3

The metric function in your case can be implemented using tf.metrics using an moving average of the loss:

def metric_fn(labels, predict):
   loss = tf.contrib.legacy_seq2seq.sequence_loss(logits, labels, weights)
   mean, op = tf.metrics.mean(loss)
   return mean, op

answered Aug 16 '17 at 19:18

Vijay Mariappan

16,921
3
40
59

I believe the [`tf.contrib.metrics.streaming_mean`](https://www.tensorflow.org/versions/r1.3/api_docs/python/tf/contrib/metrics/streaming_mean) function does this automatically. Though, it's in contrib so users beware. – Eli Bixby Aug 16 '17 at 19:22
@EliBixby isn't that function the same as `tf.metrics.mean`? i haven't checked the source but it has the same parameters and returns. – marko Dec 30 '17 at 02:26

add custom loss to eval_metric_ops

1 Answers1