Single Loss for Multiple Outputs

Question

I would like to create a neural network in Keras/Tensorflow which has multiple outputs. I would like to create a SINGLE loss function which takes all outputs into account and computes the loss accordingly. I need to do this because the outputs are related to each other. How can I achieve this? I read about concatenating all outputs to a single dense layer and then calculating the loss for this layer. Is there a more convenient way to achieve a single loss for multiple outputs yet?

I'm thinking of something like:

def my_custom_loss(y_true_0, ..., y_true_n, y_pred_0, ..., y_pred_n):
   return something

y_true_0, ..., y_true_n and y_pred_0, ..., y_pred_n

should be the true / predicted outputs of n output (dense) layers.

score 1 · Answer 1 · answered Nov 13 '19 at 13:24

You can implement loss functions based on the nature of your variables. Some standard ones are given below:

If they're just numbers (and not probabilities): MSE loss

def my_custom_loss(y_true_0, ..., y_true_n, y_pred_0, ..., y_true_n):
   y_true = tf.stack([y_true_0,...y_true_n], axis=0)
   y_pred = tf.stack([y_pred_0,...y_pred_n], axis=0)
   something = tf.losses.mean_squared_error(y_true, y_pred)
   return something

OR Absolute difference loss

def my_custom_loss(y_true_0, ..., y_true_n, y_pred_0, ..., y_true_n):
   y_true = tf.stack([y_true_0,...y_true_n], axis=0)
   y_pred = tf.stack([y_pred_0,...y_pred_n], axis=0)
   something = tf.losses.absolute_difference(y_true, y_pred)
   return something

If they're one hot vectors (valid probabilities):

def my_custom_loss(y_true_0, ..., y_true_n, y_pred_0, ..., y_true_n):
   y_true = tf.stack([y_true_0,...y_true_n], axis=0)
   y_pred = tf.stack([y_pred_0,...y_pred_n], axis=0)
   something = tf.reduce_mean(tf.keras.losses.binary_crossentropy(y_true, y_pred))
   return something

If they're zeros and ones (not valid probabilities):

def my_custom_loss(y_true_0, ..., y_true_n, y_pred_0, ..., y_true_n):
   y_true = tf.stack([y_true_0,...y_true_n], axis=0)
   y_pred = tf.stack([y_pred_0,...y_pred_n], axis=0)
   something = tf.reduce_mean(tf.keras.losses.binary_crossentropy(y_true, y_pred), from_logits=True)
   return something

It is not just limited to these. You can create your own loss function provided it is differentiable.

thanks but thats not what i was looking for. the question is how can I access multiple outputs in a single loss function. --> where do i get all the y_true_0, ..., y_true_n / y_pred_0, ..., y_pred_n from — lenngro, Nov 13 '19 at 13:29
Doesn't "I would like to create a SINGLE loss function which takes all outputs into account and computes the loss accordingly." mean this? — learner, Nov 13 '19 at 13:30
maybe I should clarify: I have a multi output network (>1 output layers). is there a way to collect all output layers logits in a single loss function — lenngro, Nov 13 '19 at 13:39
ok, now I'm confused, do you need the loss function or all the outputs? — learner, Nov 13 '19 at 13:40
sorry for confusion. i need all the outputs as inputs to a single function. usually each output layer of a network has its own loss function. — lenngro, Nov 13 '19 at 13:42

Single Loss for Multiple Outputs

1 Answers1