Keep same dataset augmentation for input and output in Tensorflow

Question

I have a batch dataset which contains image as input and output. Code is like this:

os.chdir(r'E:/trainTest')

def process_img(file_path):
    img = tf.io.read_file(file_path)
    img = tf.image.decode_png(img, channels=3)
    img = tf.image.convert_image_dtype(img, tf.float32)
    img = tf.image.resize(img, size=(img_height, img_width))
    return img

x_files = glob('input/*.png')
y_files = glob('output/*.png')

files_ds = tf.data.Dataset.from_tensor_slices((x_files, y_files))


#Dataset which gives me input-output 
files_ds = files_ds.map(lambda x, y: (process_img(x), process_img(y))).batch(batch_size)

#model init etc
#----

model.fit(files_ds,epochs=25)

Problem is I don't have enough images for my model. So my question is, how can i create augmented images (like flipped, rotated, zoomed etc) from files_ds? Because the output image has to be augmented the same way the input image is augmented.

This question actually arrived from the following question and i wanted to ask this in its own section:
Tensorflow image_dataset_from_directory for input dataset and output dataset

Hi the second image is not an output. It is the ground truth. Often referred to as Y. If you change the question in "Same augmentation for X and Y in Tensorflow" I think it's more clear. — Gino Gulamhussene, Sep 11 '20 at 07:59

Nicolas Gervais · Accepted Answer · 2020-08-25T19:12:56.843

tf.image has a bunch of random transformations you can use. For instance:

Here's an example on a completely random image of a cat.

from skimage import data
import matplotlib.pyplot as plt
import tensorflow as tf

cat = data.chelsea()

plt.imshow(cat)
plt.show()

Image with transformation:

from skimage import data
import matplotlib.pyplot as plt
import tensorflow as tf

cat = data.chelsea()

plt.imshow(tf.image.random_hue(cat, .2, .5))
plt.show()

You can implement this in you tf.data.Dataset like this:

def process_img(file_path):
    img = tf.io.read_file(file_path)
    img = tf.image.decode_png(img, channels=3)
    img = tf.image.convert_image_dtype(img, tf.float32)
    img = tf.image.resize(img, size=(img_height, img_width))
    img = tf.image.random_hue(img, 0., .5) # Here
    return img

I found a way to keep the same transformation in graph mode. It's basically to pass two images in the same call to the transformers.

import os
import tensorflow as tf
os.chdir(r'c:/users/user/Pictures')
from glob2 import glob
import matplotlib.pyplot as plt

x_files = glob('inputs/*.jpg')
y_files = glob('targets/*.jpg')

files_ds = tf.data.Dataset.from_tensor_slices((x_files, y_files))

def load(file_path):
    img = tf.io.read_file(file_path)
    img = tf.image.decode_jpeg(img, channels=3)
    img = tf.image.convert_image_dtype(img, tf.float32)
    img = tf.image.resize(img, size=(28, 28))
    return img

def process_img(file_path1, file_path2):
    img = tf.stack([load(file_path1), load(file_path2)])
    img = tf.image.random_hue(img, max_delta=.5)
    return img[0], img[1]

files_ds = files_ds.map(lambda x, y: process_img(x, y)).batch(1)

a, b = next(iter(files_ds))

plt.imshow(a[0, ...])
plt.imshow(b[0, ...])

I see, how do i concat this new augmented dataset with the original dataset though? Also I think I have to use ```seed=1``` parameter to sync input and output augmentation — Emre Özincegedik, Aug 25 '20 at 17:57

Aniket Bote · Answer 2 · 2020-08-25T18:46:52.507

2

You can use tf.keras.preprocessing.image.ImageDataGenerator for preprocessing. This is the documentation page for complete options. I have shared a small example on how you could use that with flow_from_directory. You dont have read the images beforehand and use up your RAM. The images are loaded from directory, preprocessed and fed into the model as and when required.

# we create two instances with the same arguments
data_gen_args = dict(rescale=1./255,
                     shear_range=0.2,
                     horizontal_flip=True,
                     rotation_range=90,
                     width_shift_range=0.1,
                     height_shift_range=0.1,
                     zoom_range=0.2)

image_datagen = ImageDataGenerator(**data_gen_args)
mask_datagen = ImageDataGenerator(**data_gen_args)

# Provide the same seed and keyword arguments to the fit and flow methods
seed = 1

image_generator = image_datagen.flow_from_directory(
    'data/images',
    class_mode=None,
    seed=seed)

mask_generator = mask_datagen.flow_from_directory(
    'data/masks',
    class_mode=None,
    seed=seed)

# combine generators into one which yields image and masks
train_generator = zip(image_generator, mask_generator)

model.fit(
    train_generator,
    steps_per_epoch=2000,
    epochs=50)

edited Aug 25 '20 at 18:46

answered Aug 25 '20 at 15:29

Aniket Bote

3,456
3
15
33

Thanks but my outputs are images, not labels so I can't get the output I want with this method to begin with. Does ```flow_from_directory``` support input image as well as output image? If so that would be great – Emre Özincegedik Aug 25 '20 at 16:13
I have update the ans for your Usecase. This way your output images will be augmented the same way your input images are augmented. Along with that the images will be loaded as required and wont consume your RAM. – Aniket Bote Aug 25 '20 at 16:58
I see, what does the "images" stand for in the ```image_datagen.fit(images, augment=True, seed=seed)```? – Emre Özincegedik Aug 25 '20 at 17:14
If you use `featurewise_center` or `featurewise_std_normalization` or `zca_whitening` attributes in the **ImageDataGenerator** you have to use this steps. The images stand for the loaded image data.(4d tensor). This is required to calculate the internal states. If you have huge data you can use the fit command on sample of your data. However if you do not use the above mentioned arguments you can skip this step altogether. [This](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator#fit) is the documentation page for it. – Aniket Bote Aug 25 '20 at 17:21
I think I couldn't understand. The images aren't yet loaded and will be loaded with ```flow_from_directory``` but code assumes the images are already loaded, am i missing something? Even the docs don't mention where that comes from – Emre Özincegedik Aug 25 '20 at 17:45
`mask_generator` and `image_generator` are directory iterators. The images are loaded from disk when they are required during the training process. ie while iterating in the batch. When one batch is being processed it keeps the batch ready. Therefore at any give time your RAM will only have 2 batches worth of data loaded. There is no assumption anywhere. – Aniket Bote Aug 25 '20 at 17:52
```image_datagen.fit(images, augment=True, seed=seed) NameError: name 'images' is not defined``` My problem lies here, it is not mentioned neither in the example or the docs where that variable comes from. I didn't load anything beforehand so idk what it is referring to – Emre Özincegedik Aug 25 '20 at 18:16
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/220453/discussion-between-aniket-bote-and-emre-ozincegedik). – Aniket Bote Aug 25 '20 at 18:47

Keep same dataset augmentation for input and output in Tensorflow

2 Answers2

Linked