Transforming images and masks together (Keras example)

Question

This code snippet has been taken from Keras API reference/Data Preprocessing.

Section : Example of transforming images and masks together.

link : https://keras.io/api/preprocessing/image/

# we create two instances with the same arguments
data_gen_args = dict(featurewise_center=True,
                     featurewise_std_normalization=True,
                     rotation_range=90,
                     width_shift_range=0.1,
                     height_shift_range=0.1,
                     zoom_range=0.2)
image_datagen = ImageDataGenerator(**data_gen_args)
mask_datagen = ImageDataGenerator(**data_gen_args)
# Provide the same seed and keyword arguments to the fit and flow methods
seed = 1
image_datagen.fit(images, augment=True, seed=seed)
mask_datagen.fit(masks, augment=True, seed=seed)
image_generator = image_datagen.flow_from_directory(
    'data/images',
    class_mode=None,
    seed=seed)
mask_generator = mask_datagen.flow_from_directory(
    'data/masks',
    class_mode=None,
    seed=seed)
# combine generators into one which yields image and masks
train_generator = zip(image_generator, mask_generator)
model.fit_generator(
    train_generator,
    steps_per_epoch=2000,
    epochs=50)

I understand that this snippet is augmenting both masks and images together and creating generators but I don't understand what is the image_datagen.fit(images, ... ) & mask_datagen.fit(masks, ...) doing ?

And I think here images & masks are undefined. Please explain those too.

Thank you.

Answer 1

Lets look at the documentation at fit.

As stated in there: "This computes the internal data stats related to the data-dependent transformations, based on an array of sample data. Only required if featurewise_center or featurewise_std_normalization or zca_whitening are set to True."

• Why they use it: So we need to use this only if we use featurewise_center or featurewise_std_normalization or zca_whitening in your code you have featurewise_center=True so this is the reason why they use it.
• What does it do: featurewise_center and featurewise_std_normalization centers the dataset features to have mean zero and std one. In order to do that we need to run on the data and compute this mean and std in order to normalize the data afterword. This is exactly what the fit method do. It computes those values so when you run on the data you could normalize the data accordingly.

As for images and masks they probably were defined before this snippet.

Answer 2

What are image_datagen.fit(images, ... ) & mask_datagen.fit(masks, ...) doing?

As shown in the first example of ImageDataGenerator, you need to fit the data generator in order to compute quantities required for featurewise normalization such as std, mean, and principal components if ZCA whitening is applied. fit just means calculate these properties based on the given data and store them in the ImageDataGenerator object for further use.

Here is the snippet:

(x_train, y_train), (x_test, y_test) = cifar10.load_data()
datagen = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True)

# THE IMPORTANT PART! 
# compute quantities required for featurewise normalization
# (std, mean, and principal components if ZCA whitening is applied)
datagen.fit(x_train)

Since you have two distinct datasets (images and masks), you need to perform this operation for each individual generator.

---

And I think here images & masks are undefined. Please explain those too.

You're right, they are undefined. On this case we're treating with float32 tensors of shape (batch_size, image_size[0], image_size[1], num_channels), extracted from other examples on the page), which can be obtained from a tf.data.Dataset, from loading data directly and getting only the images (removing the labels) or even from random data (using numpy).

Playing a bit with the aforementioned methods, they allow both 3D and 4D numeric tensors (one or multiple images).