I am trying to perform Mixup Augmentation (details here) but I am getting a value error as follows :
ValueError: operands could not be broadcast together with shapes (47,128,128,3) (64,1,1,1)
Following is the MixupImageDataGenerator class definition that I am using :
class MixupImageDataGenerator():
def __init__(self, generator, directory, batch_size, img_height, img_width, alpha=0.2, subset=None):
"""Constructor for mixup image data generator.
Keyword Arguments:
alpha {float} -- Mixup beta distribution alpha parameter. (default: {0.2})
subset {str} -- 'training' or 'validation' if validation_split is specified in
`generator` (ImageDataGenerator).(default: {None})
"""
self.batch_index = 0
self.batch_size = 64
self.alpha = alpha
# First iterator yielding tuples of (x, y)
self.generator1 = generator.flow_from_directory(directory,
target_size=(
128, 128),
class_mode="categorical",
batch_size=64,
shuffle=True,
subset=subset)
# Second iterator yielding tuples of (x, y)
self.generator2 = generator.flow_from_directory(directory,
target_size=(
128, 128),
class_mode="categorical",
batch_size=64,
shuffle=True,
subset=subset)
# Number of images across all classes in image directory.
self.n = self.generator1.samples
def reset_index(self):
"""Reset the generator indexes array.
"""
self.generator1._set_index_array()
self.generator2._set_index_array()
def on_epoch_end(self):
self.reset_index()
def reset(self):
self.batch_index = 0
def __len__(self):
# round up
return (self.n + self.batch_size - 1) // self.batch_size
def get_steps_per_epoch(self):
"""Get number of steps per epoch based on batch size and
number of images.
Returns:
int -- steps per epoch.
"""
return self.n // self.batch_size
def __next__(self):
"""Get next batch input/output pair.
Returns:
tuple -- batch of input/output pair, (inputs, outputs).
"""
if self.batch_index == 0:
self.reset_index()
#my_list = []
current_index = (self.batch_index * self.batch_size) % self.n
#my_list.insert(0,current_index)
#print("current index" + str(current_index))
if self.n > current_index + self.batch_size:
self.batch_index += 1
else:
self.batch_index = 0
#print("batch_index" + str(self.batch_index))
# random sample the lambda value from beta distribution.
l = np.random.beta(self.alpha, self.alpha, self.batch_size)
print("l")
print(l.shape)
X_l = l.reshape(self.batch_size, 1, 1, 1)
print("X_l")
print(X_l.shape)
y_l = l.reshape(self.batch_size, 1)
print("y_l")
print(y_l.shape)
# Get a pair of inputs and outputs from two iterators.
X1, y1 = self.generator1.next()
print("X1, y1")
print(X1.shape, y1.shape)
X2, y2 = self.generator2.next()
print("X2, y2")
print(X2.shape, y2.shape)
# Perform the mixup.
X = X1 * X_l + X2 * (1 - X_l)
y = y1 * y_l + y2 * (1 - y_l)
print("X")
print(X.shape)
print("y")
print(y.shape)
print("my_list " + str(my_list))
return X, y
def __iter__(self):
while True:
yield next(self)
Following is my code where I call the Mixup class and perform the actual augmentation :
# Create training and validation generator.
train_generator = MixupImageDataGenerator(generator=input_datagen,
directory='data/train/',
batch_size=64,
img_height=128,
img_width=128,
subset='training')
validation_generator = input_datagen.flow_from_directory(directory = 'data/train/', target_size=(128, 128), color_mode='rgb', class_mode='categorical',batch_size=64,subset='validation', shuffle=True, seed=42)
print('training steps: ', train_generator.get_steps_per_epoch())
print('validation steps: ', validation_generator.samples // batch_size)
The output:
Found 3503 images belonging to 5 classes.
Found 3503 images belonging to 5 classes.
Found 1496 images belonging to 5 classes.
training steps: 54
validation steps: 23
I initiated the training and am showing the output from the 42nd step. All the previous steps had similar outputs.
42/54 [======================>.......] - ETA: 11s - loss: 1.3245 - acc: 0.4565
current index 3392
batch_index 54
\n l
(64,)
X_l
(64, 1, 1, 1)
y_l
(64, 1)
\n X1, y1
(64, 128, 128, 3) (64, 5)
\n X2, y2
(64, 128, 128, 3) (64, 5)
\n X
(64, 128, 128, 3)
y
(64, 5)
43/54 [======================>.......] - ETA: 10s - loss: 1.3227 - acc: 0.4557
current index 3456
batch_index 0
\n l
(64,)
X_l
(64, 1, 1, 1)
y_l
(64, 1)
\n X1, y1
(47, 128, 128, 3) (47, 5)
\n X2, y2
(47, 128, 128, 3) (47, 5)
44/54 [=======================>......] - ETA: 9s - loss: 1.3233 - acc: 0.4538 current index 0
batch_index 1
\n l
(64,)
X_l
(64, 1, 1, 1)
y_l
(64, 1)
\n X1, y1
(64, 128, 128, 3) (64, 5)
\n X2, y2
(64, 128, 128, 3) (64, 5)
\n X
(64, 128, 128, 3)
y
(64, 5)
45/54 [========================>.....] - ETA: 8s - loss: 1.3214 - acc: 0.4549
current index 64
batch_index 2
\n l
(64,)
X_l
(64, 1, 1, 1)
y_l
(64, 1)
\n X1, y1
(64, 128, 128, 3) (64, 5)
\n X2, y2
(64, 128, 128, 3) (64, 5)
\n X
(64, 128, 128, 3)
y
(64, 5)
46/54 [========================>.....] - ETA: 7s - loss: 1.3211 - acc: 0.4541
current index 128
batch_index 3
\n l
(64,)
X_l
(64, 1, 1, 1)
y_l
(64, 1)
\n X1, y1
(64, 128, 128, 3) (64, 5)
\n X2, y2
(64, 128, 128, 3) (64, 5)
\n X
(64, 128, 128, 3)
y
(64, 5)
47/54 [=========================>....] - ETA: 6s - loss: 1.3193 - acc: 0.4561
current index 192
batch_index 4
\n l
(64,)
X_l
(64, 1, 1, 1)
y_l
(64, 1)
\n X1, y1
(64, 128, 128, 3) (64, 5)
\n X2, y2
(64, 128, 128, 3) (64, 5)
\n X
(64, 128, 128, 3)
y
(64, 5)
.
.
.
.
.
53/54 [============================>.] - ETA: 0s - loss: 1.3072 - acc: 0.4608
current index 576
batch_index 10
\n l
(64,)
X_l
(64, 1, 1, 1)
y_l
(64, 1)
\n X1, y1
(64, 128, 128, 3) (64, 5)
\n X2, y2
(64, 128, 128, 3) (64, 5)
\n X
(64, 128, 128, 3)
y
(64, 5)
54/54 [==============================] - 53s 986ms/step - loss: 1.3058 - acc: 0.4606 - val_loss: 1.2556 - val_acc: 0.4749
Epoch 00001: val_acc improved from -inf to 0.47486, saving model to GoDL-NP_CE_NLS_ABC-MIXUP.h5
Epoch 2/3
Thereafter, I get the following error:
ValueError: operands could not be broadcast together with shapes (47,128,128,3) (64,1,1,1)
My observations are
1) All the steps except step 43 have the following shapes :
X1, y1 (64, 128, 128, 3) (64, 5)
X2, y2 (64, 128, 128, 3) (64, 5)
But step 43 has the following shape:
X1, y1 (47, 128, 128, 3) (47, 5)
X2, y2 (47, 128, 128, 3) (47, 5)
2) Before the first step of the training has begun, the data generator seems to have already moved ahead as can be seen by the fact that cuurent_index = 124 by the time step 1 is running.
3) I feel the images available for step 43 is only 47 rather than the expected 64 (equal to batch size). I say this because for step 47, current_index = 3456 and batch_index=0. Total training images size = 3503. So, the difference is 3503 - 3456 = 47. For the next step, current index = 0 and batch_index = 1.
So, I guess what I need is the following:
- How to make sure that equal number of images are available for each step?
- How to ensure that the data generator and the training are in sync ?
