How to pass validation_data to Model.fit + Dataset?

Question

I am trying to train a simple Sequential network on generated data. I have a precomputed validation dataset.

To feed the inputs, I am using tf.data.Dataset API like suggested here: https://stackoverflow.com/a/48134765/231238

var train = Dataset.from_tensor_slices(ValueTuple.Create(trainInputs, trainOutputs));
train = train
    .repeat(2000000)
    .shuffle(buffer_size: 1024 * 8 * InterpolateBy)
    .batch(1024);
model.fit_dyn(train,
    epochs: 6*1024,
    steps_per_epoch: 4
    // line below does not work:
    , validation_data: (testInputs, testOutputs)
);

It works fine without validation_data.

If I pass validation_data as a tuple of tensors, like in the example above, e.g. (testInputs, testOutputs), it throws 'TypeError : float() argument must be a string or a number, not 'NoneType'. (This is what I used to do with train data too before switching to Dataset, and validation worked)

If I wrap testInputs and testOutputs into a Dataset similarly to the train data, e.g. Dataset.from_tensor_slices(ValueTuple.Create(testInputs, testOutputs))

I get a different error: ValueError : Error when checking input: expected sequential_input to have 2 dimensions, but got array with shape (347,).

Here 347 is the size of feature vector, hence testInputs.shape is (221, 347) and testOutputs.shape is (221, 1)

I'm not sure about your first error as I do not know the details of your data. But for the second, it shows this error because you use from_tensor_slices, which will slice your data along the first dimension. Hence a (221, 347) would be sliced in to 221 (347,) data points. If you batch your data to 221 using the batch method, it would then put it back to (221,347) shape. Note that in that case, you would need to use validation_steps argument as well, since you are using steps_per_epoch to specify no of steps you want to evaluate for train, and same should be the case for validation too. — kvish, Feb 06 '19 at 14:07
@kvish, though I did not understand your explanation completely, until trying a couple of things, simply replacing `from_tensor_slices` for _validation_ `Dataset` with `from_tensors` helped. IMHO, you should turn your comment into an answer. Still don't understand why `from_tensor_slices` works for training data, but does not for the test data. — LOST, Feb 06 '19 at 19:23

score 1 · Accepted Answer · answered Feb 06 '19 at 20:58

From our discussion, we can clarify some of the things.

First off, not very sure about the error when directly feeding it as a tuple. Might be needing more information about the data for it.

As far as feeding validation with tf data, when we use from_tensor_slices, "we create a dataset whose elements are slices of given tensors". With respect to this example, the input we are feeding is a tuple with respective shapes (221,347) and (221,1). What from_tensor_slices does is that it slices the respective numpy arrays along the 0th dimension (which is of size 221 here). The method will thus create a dataset, where each element is a tuple of shape (347,) and (1,) respectively. There will be 221 such elements in the dataset.

If we use the from_tensors method on the other hand, it creates a dataset with a single element, which comprise of the given tensors as input. So it is equivalent to directly feeding the numpy data as it is through a dataset object.

Here is a brief example of how this works for a much smaller dimension:

import numpy as np
import tensorflow as tf
np.random.seed(42)
example_train = np.random.randn(4, 4)
example_test = np.random.randn(4, 1)

print("Example Train:", example_train)
print("Example Test:", example_test)

dataset1 = tf.data.Dataset.from_tensor_slices((example_train, example_test))
dataset2 = tf.data.Dataset.from_tensors((example_train, example_test))

it1 = dataset1.make_one_shot_iterator().get_next()
it2 = dataset2.make_one_shot_iterator().get_next()

with tf.Session() as sess:
    for i in range(4):
        print("Element {} of dataset1: {}".format(i,sess.run([it1])))
    print ("Element 0 of dataset2: ", sess.run([it2]))

Result:

Example Train: [[ 0.49671415 -0.1382643   0.64768854  1.52302986]
 [-0.23415337 -0.23413696  1.57921282  0.76743473]
 [-0.46947439  0.54256004 -0.46341769 -0.46572975]
 [ 0.24196227 -1.91328024 -1.72491783 -0.56228753]]
Example Test: [[-1.01283112]
 [ 0.31424733]
 [-0.90802408]
 [-1.4123037 ]]
Element 0 of dataset1: [(array([ 0.49671415, -0.1382643 ,  0.64768854,  1.52302986]), array([-1.01283112]))]
Element 1 of dataset1: [(array([-0.23415337, -0.23413696,  1.57921282,  0.76743473]), array([0.31424733]))]
Element 2 of dataset1: [(array([-0.46947439,  0.54256004, -0.46341769, -0.46572975]), array([-0.90802408]))]
Element 3 of dataset1: [(array([ 0.24196227, -1.91328024, -1.72491783, -0.56228753]), array([-1.4123037]))]
Element 0 of dataset2:  [(array([[ 0.49671415, -0.1382643 ,  0.64768854,  1.52302986],
       [-0.23415337, -0.23413696,  1.57921282,  0.76743473],
       [-0.46947439,  0.54256004, -0.46341769, -0.46572975],
       [ 0.24196227, -1.91328024, -1.72491783, -0.56228753]]), array([[-1.01283112],
       [ 0.31424733],
       [-0.90802408],
       [-1.4123037 ]]))]

Regarding my comments about the batch method, by setting the batch_size to be 221 for putting things back together, if we change the dataset1 code to something like this and modify our printing to something like this for example:

dataset1 = tf.data.Dataset.from_tensor_slices((example_train, example_test)).batch(4)

with tf.Session() as sess:
    print ("Element 0 of dataset1: ", sess.run([it1]))

Our result:

Element 0 of dataset1:  [(array([[ 0.49671415, -0.1382643 ,  0.64768854,  1.52302986],
       [-0.23415337, -0.23413696,  1.57921282,  0.76743473],
       [-0.46947439,  0.54256004, -0.46341769, -0.46572975],
       [ 0.24196227, -1.91328024, -1.72491783, -0.56228753]]), array([[-1.01283112],
       [ 0.31424733],
       [-0.90802408],
       [-1.4123037 ]]))]

which you can see is the same as using from_tensors.

How to pass validation_data to Model.fit + Dataset?

1 Answers1