Keras `ImageDataGenerator` image and mask augments differently

Question

I'm training a semantic segmentation model using Keras with TensorFlow backend. I adopted ImageDataGenerator to do the image augmentation, including rotation, flip and shift. By following the documentation, I created a dictionary maskgen_args and used it as arguments to instantiate two ImageDataGenerator instances.

maskgen_args = dict(
    rotation_range=90,
    validation_split=VALIDATION_SPLIT
)

image_datagen = ImageDataGenerator(**maskgen_args)
mask_datagen = ImageDataGenerator(**maskgen_args)

The training data generator is done as follows, by setting seed to the same value, the mask will match the image.

training_data_generator = zip(
    image_datagen.flow_from_directory(
        data_dir,
        target_size=(512, 512),
        color_mode='rgb',
        batch_size=BATCH_SIZE,
        class_mode=None,
        save_format='jpeg',
        seed=GENERATE_SEED,
        subset='training'
    ),
    mask_datagen.flow_from_directory(
        label_dir,
        target_size=(512, 512),
        color_mode='grayscale',
        batch_size=BATCH_SIZE,
        class_mode=None,
        save_format='png',
        seed=GENERATE_SEED,
        subset='training'
    )
)

So far, there is no problem occurred. But as I need to do some extra preprocessing (eg. normalization) only for the image but not for the mask, I created another imagegen_args dictionary and used it as the arguments when instantiating the ImageDataGenerator.

maskgen_args = dict(
    rotation_range=90,
    validation_split=VALIDATION_SPLIT
)

imagegen_args = dict(
    samplewise_center=True,
    samplewise_std_normalization=True,
    channel_shift_range=10,
    brightness_range=(0.7, 1.3),
    **maskgen_args
)

image_datagen = ImageDataGenerator(**imagegen_args)
mask_datagen = ImageDataGenerator(**maskgen_args)

When I check the output of the training_data_generator, problem occurred: seems the image and mask are generated separately: they surely have random rotation, but they are rotated in different angle, unlike before. Here is an example of a food image and the mask for the food.

I checked the id of image_datagen and mask_datagen, both cases their id are different. I wonder why the first case they can rotate the image and mask with the same random angle, but not in the second case? What should I do to make them behave like the first case when I indeed need to give extra arguments to image_datagen?

score 2 · Accepted Answer · answered Nov 14 '19 at 20:00

When you set

channel_shift_range=10,
brightness_range=(0.7, 1.3)

This modifies the RNG of this generator so that the Image RNG and the Mask RNG are not in sync anymore.

I propose you use a custom Sequence for this task until the KP new API is released. (see https://github.com/keras-team/governance/blob/master/rfcs/20190729-keras-preprocessing-redesign.md)

For an example of a custom Sequence, I propose an example here: https://dref360.github.io/deterministic-da/

score 1 · Answer 2 · answered May 24 '21 at 16:46

For anyone else struggling with this - concatenating the images and masks along the channel axis is a handy way to synchronise the augmentations

image_mask = np.concatenate([image, mask], axis=3)
image_mask  = augmenter.flow(image_mask).next()
image = image_mask [:, :, :, 0]
mask = image_mask [:, :, :, 1]

Keras `ImageDataGenerator` image and mask augments differently

2 Answers2