Stack Overflow
Stack Overflow

Questions tagged [tf.dataset]

10 questions
2
votes
0 answers

How to add a nested dictionary as an input to tf.data.Dataset.from_tensor_slices

I am trying to load a dataset using the tf.data.Dataset.from_tensor_slices command. My input is a list of nested dictionaries in the following format: a_dict = { 'a' : 'blablabla', 'b' : { 'c': (tf.constant([[0.390,…
aDav
  • 41
  • 8
1
vote
1 answer

Input shape axis 0 must equal 4, got shape [5] when trying to crop image batch in tensorflow dataset pipeline

I get the following error when I try to crop a batch of images inside a tf.data.Dataset pipeline: InvalidArgumentError: Input shape axis 0 must equal 4, got shape [5] [[{{node crop_to_bounding_box/unstack}}]] [Op:IteratorGetNext] def…
evolved
  • 1,850
  • 19
  • 40
1
vote
1 answer

slow training despite using tf data pipeline

I am training image classification model which contains 21000 image. I have created data pipeline with the help of tf.data API of tensorflow. My issue is that training is too slow despite using API. I have also enabled tensorflow gpu version. Please…
Roshaan Zafar
  • 31
  • 5
0
votes
1 answer

Creating a custom TensorFlow dataset by subclassing tf.data.Dataset

Question: Is there a clean and straightforward way to create a custom dataset in TensorFlow by subclassing tf.data.Dataset, similar to the functionality available in PyTorch? Details: I'm currently working on a project that involves training deep…
Salman
  • 924
  • 1
  • 7
  • 20
0
votes
0 answers

tf.data.Dataset.from_generator long to initialize

I have a generator that I am trying to put into a tf.data.dataset. def static_syn_batch_generator( total_size: int, batch_size: int, start_random_seed:int=0, fg_seeds_ss:SampleSet=None, bg_seeds_ss:SampleSet=None,…
lr100
  • 648
  • 1
  • 9
  • 29
0
votes
1 answer

tf.data.datasets set each batch (prefetch)

I am looking for help thinking through this. I have a function (that is not a generator) that will give me any number of samples. Let's say that getting all the data I want to train (1000 samples) can't fit into memory. So I want to call this…
lr100
  • 648
  • 1
  • 9
  • 29
0
votes
1 answer

How to feed large NumPy arrays to tf.fit()

I have two NumPy arrays saved in .npy file extension. One contains x_train data and other contains y_train data. The x_train.npy file is 5.7GB of size. I can't feed it to the training by loading the whole array to the memory. Every time I try to…
Oshan Jayawardana
  • 41
  • 3
-1
votes
1 answer

How to find out the maximum length of a dimension in a ragged dataset

If I have the following dataset built from a ragged tensor, how can I get the maximum length (4 in this example) of all elements? ds = tf.data.Dataset.from_tensor_slices( tf.ragged.constant([[1, 2, 3, 4], [], [5, 6, 7], [8], []]))
Mykola Zotko
  • 15,583
  • 3
  • 71
  • 73
-1
votes
1 answer

How can I combine a py_function inside a map function?

I wanted to combine a py_function inside a map function, which took me a day, despite chatGPT's assistance. Since resizing an image with tf.image has implementation differences in relate to openCVs, I wanted to keep using the optimized tf.Dataset…
21kc
  • 21
  • 6
-2
votes
3 answers

TensorFlow | create image dataset, labelled by filename

Im trying to create Tensorflow dataset to train my model. I have a folder full of tagged photos, tagging is part of the files names. do you have a reasonable way to load the dataset for training without splitting it to different…
KobiAms
  • 11
  • 2