Keras Model for Siamese Network not Learning and always predicting the same ouput

Question

I am trying to train a Siamese neural network using Keras, with the goal of identifying if 2 images belong to same class or not. My data is shuffled and has equal number of positive examples and negative examples. My model is not learning anything and it is predicting the same output always. I am getting the same loss, validation accuracy, and validation loss every time.

Training Output

def convert(row):
    return imread(row)

def contrastive_loss(y_true, y_pred):
    margin = 1
    square_pred = K.square(y_pred)
    margin_square = K.square(K.maximum(margin - y_pred, 0))
    return K.mean(y_true * square_pred + (1 - y_true) * margin_square)

def SiameseNetwork(input_shape):
    top_input = Input(input_shape)

    bottom_input = Input(input_shape)

    # Network
    model = Sequential()
    model.add(Conv2D(96,(7,7),activation='relu',input_shape=input_shape))
    model.add(MaxPooling2D())
    model.add(Conv2D(256,(5,5),activation='relu',input_shape=input_shape))
    model.add(MaxPooling2D())
    model.add(Conv2D(256,(5,5),activation='relu',input_shape=input_shape))
    model.add(MaxPooling2D())
    model.add(Flatten())
    model.add(Dense(4096,activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(1024,activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(512,activation='relu'))
    model.add(Dropout(0.5))

    encoded_top = model(top_input)
    encoded_bottom = model(bottom_input)

    L1_layer = Lambda(lambda tensors:K.abs(tensors[0] - tensors[1]))    
    L1_distance = L1_layer([encoded_top, encoded_bottom])

    prediction = Dense(1,activation='sigmoid')(L1_distance)
    siamesenet = Model(inputs=[top_input,bottom_input],outputs=prediction)
    return siamesenet

data = pd.read_csv('shuffleddata.csv')
print('Converting X1....')

X1 = [convert(x) for x in data['X1']]

print('Converting X2....')

X2 = [convert(x) for x in data['X2']]

print('Converting Y.....')
Y = [0 if data['Y'][i] == 'Negative' else 1 for i in range(len(data['Y']))]

input_shape = (53,121,3,)
model = SiameseNetwork(input_shape)
model.compile(loss=contrastive_loss,optimizer='sgd',metrics=['accuracy'])
print(model.summary())
model.fit(X1,Y,batch_size=32,epochs=20,shuffle=True,validation_split = 0.2)
model.save('Siamese.h5')

Answer 1

Mentioning the resolution to this issue in this section (even though it is present in Comments Section), for the benefit of the community.

Since the Model is working fine with other Standard Datasets, the solution is to use more Data. Model is not learning because it has less data for Training.

Answer 2

The Model is working fine with more data as mentioned in comments and in the answer by Tensorflow Support. Tweaking the model a little is also working. Changing the number of filters in 2nd and 3rd convolutional layers from 256 to 64 is decreasing the number of trainable parameters by a large number and therefore model started learning.

Answer 3

I want to mention few things here which may be useful to others:

1) Data stratification / random sampling

When you use validation_split Keras uses the last x percent of data as validation data. This means that if the data is ordered by class, e.g. because "pairs" or "tripletts" are made in a sequence, validation data will only come from classes (or the class) contained in the last x percent of data. In this case, the validation set will be of no use. Thus it is essential to suffle input data to make sure that the validation set contains random samples from each class.

The docs for validation_split say:

Float between 0 and 1. Fraction of the training data to be used as validation data. The model will set apart this fraction of the training data, will not train on it, and will evaluate the loss and any model metrics on this data at the end of each epoch. The validation data is selected from the last samples in the x and y data provided, before shuffling

2) Choice of optimizer

In model.compile() choosing optimizer='sgd' may not be the best approach since sgd can get stuck in local minima etc. Adam (see docs) seems to be a good choice to start with since it...

[...] combines the advantages of [...] AdaGrad to deal with sparse gradients, and the ability of RMSProp to deal with non-stationary objectives.

according to Kingma and Ba (2014, page 10).

from keras.optimizers import Adam
...
model.compile(loss=contrastive_loss, optimizer=keras.optimizers.Adam(lr=0.0001))

3) Early stopping / learning rate

Using early stopping and adjusting the learning rate during training may also be highly useful to achieve good results. So the model can train until there is no more success (stop automatically in this case).

from keras.callbacks import EarlyStopping
from keras.callbacks import ReduceLROnPlateau
...
early_stopping = EarlyStopping(monitor='val_loss', patience=50, mode='auto', restore_best_weights=True)
reduce_on_plateau = ReduceLROnPlateau(monitor="val_loss", factor=0.8, patience=15, cooldown=5, verbose=0)
...
hist = model.fit([img_1, img_2], y, 
            validation_split=.2, 
            batch_size=128, 
            verbose=1, 
            epochs=9999,
            callbacks=[early_stopping])

4) Kernel initialization

Kernel initialization (with a small SD) may be helpful as well.

# Layer 1
    seq.add(Conv2D(8, (5,5), input_shape=input_shape, 
        kernel_initializer=keras.initializers.TruncatedNormal(mean=0.0, stddev=0.01, seed=None), 
        data_format="channels_first"))
    seq.add(Activation('relu'))
    seq.add(MaxPooling2D(pool_size=(2, 2))) 
    seq.add(Dropout(0.1))

5) Overfitting

I noticed that instead of using dropout to fight overfitting, adding some noise can be rather helpful. In this case simply add some GaussianNoise at the top of the network.