2

There are several tutorials that applied reduce_mean to the output of sparse_softmax_cross_entropy_with_logits. For example

cross_entropy = -tf.reduce_sum(y_ * tf.log(y_conv))

or 

cross_entropy = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
                               labels=tf.cast(y_, dtype=tf.int32), logits=y_conv))

Why is the reduce_mean applied to the output of sparse_softmax_cross_entropy_with_logits? Is it because we are using mini-batches, and so we want to calculate (using reduce_mean) the average loss over all samples of the mini-batch?

nbro
  • 15,395
  • 32
  • 113
  • 196
Hong Cheng
  • 318
  • 1
  • 4
  • 19

2 Answers2

4

The reason is to get the average loss over the batch.

Generally you will train a neural network with input batches of size > 1, each element in the batch will produce a loss value so the easiest way to merge these into one value is to average.

Burton2000
  • 1,972
  • 13
  • 23
0

I find something interesting~

first, let define sparse_vector as

sparse_vector = tf.nn.sparse_softmax_cross_entropy_with_logits(
            labels=tf.cast(y_, dtype=tf.int32), logits=y_conv)

the sparse_vector is a vector, and we should calculate the summery of it, that why we should use the reduce_mean.

import numpy as np
import tensorflow as tf

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.InteractiveSession(config=config)

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data', one_hot=False)

print(mnist.test.labels.shape)
print(mnist.train.labels.shape)

with tf.name_scope('inputs'):
    X_ = tf.placeholder(tf.float32, [None, 784])
    y_ = tf.placeholder(tf.int64, [None])

X = tf.reshape(X_, [-1, 28, 28, 1])
h_conv1 = tf.layers.conv2d(X, filters=32, kernel_size=5, strides=1, 
                           padding='same', activation=tf.nn.relu, name='conv1')
h_pool1 = tf.layers.max_pooling2d(h_conv1, pool_size=2, strides=2,
                                  padding='same', name='pool1')

h_conv2 = tf.layers.conv2d(h_pool1, filters=64, kernel_size=5, strides=1, 
                           padding='same',activation=tf.nn.relu, name='conv2')
h_pool2 = tf.layers.max_pooling2d(h_conv2, pool_size=2, strides=2, 
                                  padding='same', name='pool2')

# flatten
h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
h_fc1 = tf.layers.dense(h_pool2_flat, 1024, name='fc1', activation=tf.nn.relu)


keep_prob = tf.placeholder(tf.float32)
h_fc1_drop = tf.nn.dropout(h_fc1, 0.5)  

h_fc2 = tf.layers.dense(h_fc1_drop, units=10, name='fc2')
# y_conv = tf.nn.softmax(h_fc2)
y_conv = h_fc2
# print('Finished building network.')

# cross_entropy = -tf.reduce_sum(y_*tf.log(y_conv))
sparse_vector = tf.nn.sparse_softmax_cross_entropy_with_logits(
            labels=tf.cast(y_, dtype=tf.int32), logits=y_conv)
cross_entropy = tf.reduce_mean(sparse_vector)
sess.run(tf.global_variables_initializer())
# print(sparse_vector)
# print(cross_entropy)
# Tensor("SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits:0", shape=(?,), dtype=float32)
# Tensor("Mean:0", shape=(), dtype=float32)

batch = mnist.train.next_batch(10)
sparse_vector,cross_entropy = sess.run(
    [sparse_vector,cross_entropy],
    feed_dict={X_: batch[0], y_: batch[1]})

print(sparse_vector)
print(cross_entropy)

the output is [2.2213464 2.2676413 2.3555744 2.3196406 2.0794516 2.394274 2.266591 2.3139718 2.345526 2.3952296]

2.2959247

Hong Cheng
  • 318
  • 1
  • 4
  • 19