Tensorflow Data Input Toggle: Train/Validation

Question

I have data that comes into my graph through queue runners, after I switched from the handy but speed-inferior placeholders.

After each training epoch, I wish to run a validation pass. Other than the the training pass, the validation pass uses different data, no augmentation and no shuffling.

The question is simple: how do I toggle these things?

A few observations:

• I cannot toggle the shuffle option in the string_input_producer through a tf.placeholder boolean.
• The only examples on-the-line that I have found use the placeholder to seperate the training from the validation data. These in turn, do not use the superior queue runners.
• I did manage to do the above with a tf.cond() here i would test for a is_training tf.placeholder boolean that i pass through the feed_dict. Is this solution the most optimal? How expensive is this tf.conf() method?

Answer 1

The method that works well for me is to use tf.placeholder_with_default:

images_train, labels_train = train_data_pipeline(fnlist_train, ref_grid)
images_val, labels_val = val_data_pipeline(fnlist_val, ref_grid)
images = tf.placeholder_with_default(images_train, shape=[None, FLAGS.nx_image, FLAGS.ny_image, FLAGS.nz_image])
labels = tf.placeholder_with_default(labels_train, shape=[None, label_length])

During training, images and labels come directly from the training queue. For the intermittent validation steps I feed images and labels through a feed_dict in a call tosess.run(). The only slight hack is that is that the validation data are also tensors from a queue and feed_dict doesn't accept tensors, so I call sess.run([images_val, labels_val]) first to get numpy values and then use them in the feed_dict. Seems to work well and there is minimal delay from the tensor==>numpy==>tensor conversion, which only occurs during validation anyway.

And for your case where the validation data have separate processing requirements, this can be handled when you set up the separate validation queue and processing flow to it.

Answer 2

One probable answer is usage of make_template This is outlined in https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/kernel_tests/template_test.py ; It basically says one could do this:

training_input, training_output = ([1., 2., 3., 4.], [2.8, 5.1, 7.2, 8.7])
test_input, test_output = ([5., 6., 7., 8.], [11, 13, 15, 17])

tf.set_random_seed(1234)

def test_line(x):
  m = tf.get_variable("w", shape=[],
                      initializer=tf.truncated_normal_initializer())
  b = tf.get_variable("b", shape=[],
                      initializer=tf.truncated_normal_initializer())
  return x * m + b

line_template = template.make_template("line", test_line)

train_prediction = line_template(training_input)
test_prediction = line_template(test_input)

train_loss = tf.reduce_mean(tf.square(train_prediction - training_output))
test_loss = tf.reduce_mean(tf.square(test_prediction - test_output))

optimizer = tf.train.GradientDescentOptimizer(0.1)
train_op = optimizer.minimize(train_loss)

with tf.Session() as sess:
  sess.run(tf.initialize_all_variables())
  initial_test_loss = sess.run(test_loss)
  sess.run(train_op)
  final_test_loss = sess.run(test_loss)

# Parameters are tied, so the loss should have gone down when we trained it.
self.assertLess(final_test_loss, initial_test_loss)