0

I'm running a FCN in Keras that uses the binary cross-entropy as the loss function. However, im not sure how the losses are accumulated.

I know that the loss gets applied at the pixel level, but then are the losses for each pixel in the image summed up to form a single loss per image? Or instead of being summed up, is it being averaged?

And furthermore, are the loss of each image simply summed(or is it some other operation) over the batch?

Jonathan
  • 1,876
  • 2
  • 20
  • 56

1 Answers1

1

I assume that you question is a general one, and to specific to a particular model (if not can you share your model?).

You are right that if the cross-entropy is used at a pixel level, the results have to be reduced (summed or averaged) over all pixels to get a single value.

Here is an example of a convolutional autoencoder in tensorflow where this step is specific:

https://github.com/udacity/deep-learning/blob/master/autoencoder/Convolutional_Autoencoder_Solution.ipynb

The relevant lines are:

loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=targets_, logits=logits)
cost = tf.reduce_mean(loss)

Whether you take the mean or sum of the cost function does not change the value of the minimizer. But If you take the mean, then the value of the cost function is more easily comparable between experiments when you change the batch size or image size.

Alex338207
  • 1,825
  • 12
  • 16
  • Its just a general one, not related to a specific model. But specific to keras – Jonathan Aug 22 '18 at 07:13
  • So my question is specifically is it summed or averaged in keras – Jonathan Aug 22 '18 at 07:14
  • In Keras, the cost function is summed: https://github.com/keras-team/keras/blob/5a7a789ee9766b6a594bd4be8b9edb34e71d6500/keras/backend/tensorflow_backend.py#L3213 – Alex338207 Aug 22 '18 at 07:47
  • So does that mean the model wont take into account individual image loss? Consider the following scenario, all the pixel losses are added per image, then the loss per image is summed for all the images in that batch. If i had 16 image(256×256) batch sizes, then loss over one iteration will be 16x256x256 individual loss calculations summed, and the loss will never take into account individual images unless the batch size is 1. Am i understanding this right? – Jonathan Aug 22 '18 at 16:09
  • The total loss over the batch size is essentially the sum of the loss of each image in your batch size (if I understand you question). – Alex338207 Aug 23 '18 at 09:51
  • I was asking if the loss per image is the mean or summation of all the pixels in the image when using binary cross entropy. The documentation says `k.mean(binaryCrossentropy) ` but im unable to confirm if the binaryCrossentropy actually works at the pixel level looking at the code – Jonathan Aug 23 '18 at 15:11
  • Even though pixels are not IID, we make the hypothesis that they are. This means your exemples are pixels, not images. Thus, from a probabilistic point of view, you should sum the error per image and divide by the total number of pixels. Micro-average. – cortax Jun 11 '20 at 14:07