Tensorflow Optimizers - multiple loss values passed to minimize()?

Question

My first time using Tensorflow on the MNIST dataset, I had a really simple bug where I forgot to take mean of my error values before passing it to the optimizer.

In other words, instead of

loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y, labels=y_))

I accidentally used

loss = tf.nn.softmax_cross_entropy_with_logits(logits=y, labels=y_)

Not taking the mean or sum of the error values threw no errors when training the network, however. This led me thinking: Is there actually a case when someone would need to pass in multiple loss values into an optimizer? What was happening when I passed in a Tensor not of size [1] into minimize()?

score 4 · Answer 1 · answered May 12 '17 at 01:42

4

They are being added up. This is side-product of TensorFlow using Reverse Mode AD to differentiate, which requires loss to be a scalar

answered May 12 '17 at 01:42

Yaroslav Bulatov

57,332
22
139
197

So if you pass a tensor, you basically minimizing the sum of all values in the tensor? – Salvador Dali May 12 '17 at 02:31
2

;;;;;;;;;;;;;;;;;;;;yes – Yaroslav Bulatov May 12 '17 at 03:13

Tensorflow Optimizers - multiple loss values passed to minimize()?

1 Answers1