5

I am trying to use Tensorboard to visualize my training procedure. My purpose is, when every epoch completed, I would like to test the network's accuracy using the whole validation dataset, and store this accuracy result into a summary file, so that I can visualize it in Tensorboard.

I know Tensorflow has summary_op to do it, however it seems only work for one batch when running the code sess.run(summary_op). I need to calculate the accuracy for the whole dataset. How?

Is there any example to do it?

C. Wang
  • 2,516
  • 5
  • 29
  • 46

3 Answers3

9

Define a tf.scalar_summary that accepts a placeholder:

accuracy_value_ = tf.placeholder(tf.float32, shape=())
accuracy_summary = tf.scalar_summary('accuracy', accuracy_value_)

Then calculate the accuracy for the whole dataset (define a routine that calculates the accuracy for every batch in the dataset and extract the mean value) and save it into a python variable, let's call it va.

Once you have the value of va, just run the accuracy_summary op, feeding the accuracy_value_ placeholder:

sess.run(accuracy_summary, feed_dict={accuracy_value_: va})
nessuno
  • 26,493
  • 5
  • 83
  • 74
  • 1
    I really like the proposed solution as this is exactly what I want to achieve, and not what streaming metrics push me to do – Tosha Jun 15 '18 at 11:14
  • I'm glad it helped! – nessuno Jun 15 '18 at 11:27
  • But in some cases, the accuracy of the whole dataset is not precisely the mean of the accuracy of each batch, e.g., there is 1000 data, and `batch_size=128`. The last batch size is not 128 so it can not simply apply average. – huangbiubiu Jun 18 '18 at 14:19
  • 1
    @HuangYuheng in that case you can use a weighted mean: weight the accuracy at each iteration by the number of elements used to compute the accuracy, then divide by the sum of the weights (which will be the length of the complete dataset). – Rohan Saxena Jul 11 '18 at 09:38
  • @nessuno, Thanks for your answer, and I believe it is so close to what I want. However, for training and validation, the value of summary comes from different calculation path, for example, in training, the value should be directly a loss instead of a placeholder. So how can I use a single node for summary operation but it can handle both ways of calculations? – C. Wang Mar 18 '20 at 14:26
  • No, if you define this node to accept a placeholder you must feed a value to that placeholder when you want to write on it. If you chose this option, you have to do first a loss_value = sess.run(loss), and the feeding the placeholder accuracy_value with loss_value – nessuno Mar 18 '20 at 15:07
0

I implement a naive one-layer model as an example to classify MNIST dataset and visualize validation accuracy in Tensorboard, it works for me.

import tensorflow as tf
from tensorflow.contrib.learn.python.learn.datasets.mnist import read_data_sets
import os

# number of epoch
num_epoch = 1000
model_dir = '/tmp/tf/onelayer_model/accu_info'
# mnist dataset location, change if you need
data_dir = '../data/mnist'

# load MNIST dataset without one hot
dataset = read_data_sets(data_dir, one_hot=False)

# Create placeholder for input images X and labels y
X = tf.placeholder(tf.float32, [None, 784])
# one_hot = False
y = tf.placeholder(tf.int32)

# One layer model graph
W = tf.Variable(tf.truncated_normal([784, 10], stddev=0.1))
b = tf.Variable(tf.constant(0.1, shape=[10]))
logits = tf.nn.relu(tf.matmul(X, W) + b)

init = tf.initialize_all_variables()

cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits, y)
# loss function
loss = tf.reduce_mean(cross_entropy)
train_op = tf.train.GradientDescentOptimizer(0.01).minimize(loss)

_, top_1_op = tf.nn.top_k(logits)
top_1 = tf.reshape(top_1_op, shape=[-1])
correct_classification = tf.cast(tf.equal(top_1, y), tf.float32)
# accuracy function
acc = tf.reduce_mean(correct_classification)

# define info that is used in SummaryWritter
acc_summary = tf.scalar_summary('valid_accuracy', acc)
valid_summary_op = tf.merge_summary([acc_summary])

with tf.Session() as sess:
    # initialize all the variable
    sess.run(init)

    print("Writing Summaries to %s" % model_dir)
    train_summary_writer = tf.train.SummaryWriter(model_dir, sess.graph)

    # load validation dataset
    valid_x = dataset.validation.images
    valid_y = dataset.validation.labels

    for epoch in xrange(num_epoch):
        batch_x, batch_y = dataset.train.next_batch(100)
        feed_dict = {X: batch_x, y: batch_y}
        _, acc_value, loss_value = sess.run(
            [train_op, acc, loss], feed_dict=feed_dict)
        vsummary = sess.run(valid_summary_op,
                            feed_dict={X: valid_x,
                                       y: valid_y})

        # Write validation accuracy summary
        train_summary_writer.add_summary(vsummary, epoch)
xiaoming-qxm
  • 1,738
  • 14
  • 23
0

Using batching with your validation set is possible in case you are using tf.metrics ops, which use internal counters. Here is a simplified example:

model = create_model()
tf.summary.scalar('cost', model.cost_op)
acc_value_op, acc_update_op = tf.metrics.accuracy(labels,predictions)

summary_common = tf.summary.merge_all()

summary_valid = tf.summary.merge([
    tf.summary.scalar('accuracy', acc_value_op),
    # other metrics here...
])

with tf.Session() as sess:
    train_writer = tf.summary.FileWriter(logs_path + '/train',
                                         sess.graph)
    valid_writer = tf.summary.FileWriter(logs_path + '/valid')

While training, only write the common summary using your train-writer:

summary = sess.run(summary_common)
train_writer.add_summary(summary, tf.train.global_step(sess, gstep_op))
train_writer.flush()

After every validation, write both summaries using the valid-writer:

gstep, summaryc, summaryv = sess.run([gstep_op, summary_common, summary_valid])
valid_writer.add_summary(summaryc, gstep)
valid_writer.add_summary(summaryv, gstep)
valid_writer.flush()

When using tf.metrics, don't forget to reset the internal counters (local variables) before every validation step.

b3nk4n
  • 1,101
  • 1
  • 11
  • 17