Is there a way of knowing if `.repeat`/`.batch`/`.shuffle` have been used on a tensorflow dataset?

Question

One gets an already built tensorflow dataset object (tf.data.Dataset) named data.

Is there a way to know if the function repeat/batch/shuffle was called on this object, by inspecting data ? (and possibly get other informations like the argument of batch and repeat)

(I assume eager execution)

edit 1: seems line the str method carries some information. Looking into that.

edit 2: the attribute output_shapes give information on the batch size and shapes.

define "by inspecting data ?" You mean by looking at the output? — Sharky, Mar 28 '19 at 17:55
I mean by looking at the attributes/methods of the object data — jeandut, Mar 28 '19 at 18:06

Sharky · Answer 1 · 2019-03-28T19:08:48.910

0

The only solution I could think of is getting into tensorflow code. gen_dataset_ops.py is generated during building from source, so it could only be found locally.

Another file is dataset_ops.py, it's available in the link below. You just insert print statement before relevant function's return. For example shuffle function from dataset_ops.py:

def shuffle(self, buffer_size, seed=None, reshuffle_each_iteration=None):
"""Randomly shuffles the elements of this dataset.
...
print('Dataset shuffled') #inserted print here
return ShuffleDataset(self, buffer_size, seed, reshuffle_each_iteration)

Dataset object is wrapped into DatasetV1Adapter, so you can't know anything about it advance. The only difference in eager mode is that it supports explicit iteration, but it'll be extremely inefficient to do smth like

array = np.random.rand(10)
dataset = tf.data.Dataset.from_tensor_slices(array)
if len([i for i in dataset]) != array.shape[0]:
    print('repeated')

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/data/ops/dataset_ops.py

edited Mar 28 '19 at 19:08

answered Mar 28 '19 at 18:20

Sharky

4,473
2
19
27

We don't have access to array – jeandut Mar 28 '19 at 18:33
We only have the dataset object – jeandut Mar 28 '19 at 18:34
This is more a counter example. I updated answer, this is rough but it works – Sharky Mar 28 '19 at 18:53
I think you should remove in your answer the part about the eager mode. It is misleading. – jeandut Mar 28 '19 at 19:06
Maybe it comes from my question not being explicit. – jeandut Mar 28 '19 at 19:07
Should be clear now. eager mode doesn't matter in this case – Sharky Mar 28 '19 at 19:10

Is there a way of knowing if `.repeat`/`.batch`/`.shuffle` have been used on a tensorflow dataset?

1 Answers1