I am trying to reuse the PTB language model on my data but lacking knowledge of Tensorflow to understand how does it handle batch iteration over the training data. Here is how I understand batch iteration during training:
while epoch <= maxepoch do
for minibatch in data_iterator() do
model.forward(minibatch)
(...)
end
end
Cannot get simpler than this, can it? Something similar is done in many other frameworks but not in Tensorflow :) Here is a sample of minibatch function from official PTB language model tutorial:
def ptb_producer(raw_data, batch_size, num_steps, name=None):
with tf.name_scope(name, "PTBProducer", [raw_data, batch_size, num_steps]):
raw_data = tf.convert_to_tensor(raw_data, name="raw_data", dtype=tf.int32)
data_len = tf.size(raw_data)
batch_len = data_len // batch_size
data = tf.reshape(raw_data[0 : batch_size * batch_len],
[batch_size, batch_len])
epoch_size = (batch_len - 1) // num_steps
assertion = tf.assert_positive(
epoch_size,
message="epoch_size == 0, decrease batch_size or num_steps")
with tf.control_dependencies([assertion]):
epoch_size = tf.identity(epoch_size, name="epoch_size")
i = tf.train.range_input_producer(epoch_size, shuffle=False).dequeue()
x = tf.strided_slice(data, [0, i * num_steps], [batch_size, (i + 1) * num_steps])
x.set_shape([batch_size, num_steps])
y = tf.strided_slice(data, [0, i * num_steps + 1], [batch_size, (i + 1) * num_steps + 1])
y.set_shape([batch_size, num_steps])
return x, y
This function returns x
inputs and y
targets once it is called. I see no signs of Python iterator here but there is a call to tf.strided_slice
which uses i
index generated by tf.train.range_input_producer
so this should emulate a sliding window over the data. However the function is called only once before the training so how can it iterate over my data then? This is unclear. Can somebody explain this "magic" and completely obscure Tensorflow mechanism?