Data generator for image classification

Question

I'm trying to create a data generator for my CNN project(using a sequential model in keras). Due to the large amount of data, I need to keep flowing data to the model training so I dont get OOM on RAM. However, I'm having some trouble with creating the generator. The generator should take in the batch_size of data and then create X number of augmented images. Then I want to create a batch of the created augmented images and the original, e.g 30 original pictures, 5 augmented images per images = 30 original pictures + 150 augmented pictures = 180 total pictures in one batch. I then want to take a batch_size from this 180 pictures, lets say 30, this will create 6 epoch steps with 30 images per step. Then i want to generate a new batch of images and repeat these steps for X amount of epochs.

Code:

class customDataGen(tf.keras.utils.Sequence):
    data_holder_x = []
    data_holder_y = []
    
    ## leave out img_gen, that does not do anything right now.
    def __init__(self, X, y, img_gen, batch_size, shuffle = True):
        self.X = X
        self.y = y
        self.batch_size = batch_size
        self.shuffle = shuffle
        self.img_gen = img_gen
        
        nr1 = 5*self.batch_size ## The image augmentation does generates 5 images per image so im just hard-coding in 5 right now.
        nr2 = self.batch_size ## this is the original pictures
        self.n = nr1 + nr2
        self.indices = list(range(0,self.n))
        self.__get_data(index=1) ## just generating a instance of get_data 
        
    
    def on_epoch_end(self):
        self.index = np.arange(len(self.indices))
        if self.shuffle == True:
            np.random.shuffle(self.index)
    
    def __get_data(self,index):
        print("get_data startad")
        aug_img = img_aug(self.X[index*self.batch_size:(index+1)*self.batch_size],self.y[index*self.batch_size:(index+1)*self.batch_size])
        X = list(self.X[index*self.batch_size:(index+1)*self.batch_size])
        y = list(self.y[index*self.batch_size:(index+1)*self.batch_size])                  
        X.extend(aug_img[0])
        y.extend(aug_img[1])
        customDataGen.data_holder_x.append(X)
        customDataGen.data_holder_y.append(y)
    
    def __data_holder(self,index):
        container_x = []
        container_y = []
        print("__data_holder startad")
        if len(customDataGen.data_holder_x[0]) == 0:
            self.__get_data(index)
            container_x.append(customDataGen.data_holder_x[0][:self.batch_size])
            container_y.append(customDataGen.data_holder_y[0][:self.batch_size])
            del customDataGen.data_holder_x[0][:self.batch_size], customDataGen.data_holder_y[0][:self.batch_size]
        else:
            container_x.append(customDataGen.data_holder_x[0][:self.batch_size])
            container_y.append(customDataGen.data_holder_y[0][:self.batch_size])
            del customDataGen.data_holder_x[0][:self.batch_size], customDataGen.data_holder_y[0][:self.batch_size]
        #X = np.array(container_x[0][0])
        #y = np.array(container_y[0][0])
        print("remaining data of data_holder_x", len(customDataGen.data_holder_x[0]))
        return container_x[0],container_y[0]
        
    def __getitem__(self,index):
        container_x,container_y = self.__data_holder(index)
        print("get_item startad")
        X = tf.convert_to_tensor(container_x)
        y = tf.convert_to_tensor(container_y)
        return X,y
    
    def __len__(self):
        return (self.n)//self.batch_size

My problem now is that it seems like __get_item is getting called and initiated by model.fit() 3 times before the epoch start

__data_holder startad
remaining data of data_holder_x 160
get_item startad
Epoch 1/2
__data_holder startad
remaining data of data_holder_x 128
get_item startad
__data_holder startad
remaining data of data_holder_x 96
get_item startad
1/6 [====>.........................] - ETA: 15s - loss: 1.7893 - accuracy: 0.1562__data_holder startad
remaining data of data_holder_x 64
get_item startad
2/6 [=========>....................] - ETA: 6s - loss: 1.7821 - accuracy: 0.2344 __data_holder startad
remaining data of data_holder_x 32
get_item startad
3/6 [==============>...............] - ETA: 4s - loss: 1.7879 - accuracy: 0.1562__data_holder startad
remaining data of data_holder_x 0
get_item startad
4/6 [===================>..........] - ETA: 3s - loss: 1.7878 - accuracy: 0.1953__data_holder startad
get_data startad
remaining data of data_holder_x 0
get_item startad
5/6 [========================>.....] - ETA: 1s - loss: 1.7888 - accuracy: 0.1875

Then the error occurs

2022-09-30 17:44:31.255235: W tensorflow/core/framework/op_kernel.cc:1733] INVALID_ARGUMENT: TypeError: `generator` yielded an element of shape (0,) where an element of shape (None, None, None, None) was expected.
Traceback (most recent call last):

  File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/ops/script_ops.py", line 270, in __call__
    ret = func(*args)

  File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/autograph/impl/api.py", line 642, in wrapper
    return func(*args, **kwargs)

  File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/data/ops/dataset_ops.py", line 1073, in generator_py_func
    raise TypeError(

TypeError: `generator` yielded an element of shape (0,) where an element of shape (None, None, None, None) was expected.


---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
Input In [298], in <cell line: 1>()
----> 1 model.fit(training,
      2           validation_data=validation,
      3           epochs=2, callbacks = [checkpoint])

File /usr/local/lib/python3.9/dist-packages/keras/utils/traceback_utils.py:67, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     65 except Exception as e:  # pylint: disable=broad-except
     66   filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67   raise e.with_traceback(filtered_tb) from None
     68 finally:
     69   del filtered_tb

File /usr/local/lib/python3.9/dist-packages/tensorflow/python/eager/execute.py:54, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     52 try:
     53   ctx.ensure_initialized()
---> 54   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     55                                       inputs, attrs, num_outputs)
     56 except core._NotOkStatusException as e:
     57   if name is not None:

InvalidArgumentError: Graph execution error:

2 root error(s) found.
  (0) INVALID_ARGUMENT:  TypeError: `generator` yielded an element of shape (0,) where an element of shape (None, None, None, None) was expected.
Traceback (most recent call last):

  File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/ops/script_ops.py", line 270, in __call__
    ret = func(*args)

  File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/autograph/impl/api.py", line 642, in wrapper
    return func(*args, **kwargs)

  File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/data/ops/dataset_ops.py", line 1073, in generator_py_func
    raise TypeError(

TypeError: `generator` yielded an element of shape (0,) where an element of shape (None, None, None, None) was expected.


     [[{{node PyFunc}}]]
     [[IteratorGetNext]]
     [[IteratorGetNext/_2]]
  (1) INVALID_ARGUMENT:  TypeError: `generator` yielded an element of shape (0,) where an element of shape (None, None, None, None) was expected.
Traceback (most recent call last):

  File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/ops/script_ops.py", line 270, in __call__
    ret = func(*args)

  File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/autograph/impl/api.py", line 642, in wrapper
    return func(*args, **kwargs)

  File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/data/ops/dataset_ops.py", line 1073, in generator_py_func
    raise TypeError(

TypeError: `generator` yielded an element of shape (0,) where an element of shape (None, None, None, None) was expected.


     [[{{node PyFunc}}]]
     [[IteratorGetNext]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_5083]

Im new to both python and tensorflow so any help is appreciated.

Thanks,

Pythonnorra

elbe · Answer 1 · 2022-10-01T00:11:30.483

1

With data augmentation, the Sequence class is not really simple to use. It seems to call getitem when building the graph before the first epoch. A good and interesting alternative to the Sequence class is the use of tensorflow tf.data.Dataset

Here is some dummy code that does the job :

mydataset = tf.data.Dataset.from_tensor_slices((X, y))
def aug_data(x, y):
  # dummy data augmentation
  x_aug = tf.repeat(x, 5, axis=0)
  y_aug = tf.repeat(y, 5, axis=0)
  # concatenate with original data
  x_aug = tf.concat([x, x_aug], axis=0)
  y_aug = tf.concat([y, y_aug], axis=0)
  return x_aug, y_aug

# apply data augmentation on batches
mydataset = mydataset.map(aug_data).batch(BATCH_SIZE)

It applies the data augmentation to each new batch individually.

edited Oct 01 '22 at 00:11

answered Sep 30 '22 at 23:38

elbe

1,363
1
9
13

Thanks for your comment. However, im calling a img_aug function that use the library albumentations. Each image that gets passed to img_aug() will be augmented X times. I then need to add the augmented pictures with the original. How is this possible with this answer? Ive been trying to develop this custom data generator for days now and yes, it seems like __get_item_ in the sequence gets called multiple times to build the graph. With your recommendation to use tf.data.Dataset, there is a problem.. This will convert the original pictures to tensors? – Pythonnorra Oct 02 '22 at 14:22
As I understand it, using a Sequence subclass, in `__getitem__`, you could take `batch_size // (X+1)` original images and pass each of image trough img_aug() and append to the original image. This process would generate a complete batch. – elbe Oct 02 '22 at 17:51

Data generator for image classification

1 Answers1