Validation Loss and Validation Accuracy Curve Fluctuating with the Pretrained Model

Question

I'm currently studying about neural network and I'm having problem when try to learn about CNN, I'm trying to train data which contains Spectrogram about music genres. My data consists of 27000 Spectrogram and divided into 3 class (genre). My data splitted in 9:1 ratio for training and validation

Can anyone help me why is the result for my validation loss/accuracy is fluctuating? I'm using MobileNetV2 from Keras and connect it with 3 Dense layers. Here is the snippet of my codes:

train_datagen = ImageDataGenerator(
    preprocessing_function=preprocess_input,
    validation_split=0.1)

train_generator = train_datagen.flow_from_dataframe(
    dataframe=traindf,
    directory="...",
    color_mode='rgb',
    x_col="ID",
    y_col="Class",
    subset="training",
    batch_size=32,
    seed=42,
    shuffle=True,
    class_mode="categorical",
    target_size=(64, 64))

valid_generator = train_datagen.flow_from_dataframe(
    dataframe=traindf,
    directory="...",
    color_mode='rgb',
    x_col="ID",
    y_col="Class",
    subset="validation",
    batch_size=32,
    seed=42,
    shuffle=True,
    class_mode="categorical",
    target_size=(64, 64))

base_model = MobileNetV2(weights='imagenet', include_top=False)
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1025, activation='relu')(x)
x = Dense(1025, activation='relu')(x)
x = Dense(512, activation='relu')(x)
preds = Dense(3, activation='softmax')(x)
model = Model(inputs=base_model.input, outputs=preds)

model.compile(optimizer='adam', loss='categorical_crossentropy',
                  metrics=['accuracy'])

step_size_train = train_generator.n//train_generator.batch_size
step_size_valid = valid_generator.n//valid_generator.batch_size
history = model.fit_generator(
    generator=train_generator,
    steps_per_epoch=step_size_train,
    validation_data=valid_generator,
    validation_steps=step_size_valid,
    epochs=75)

These are the picture of my validation loss and validation accuracy curve that fluctuating too much

Is there anyway to reduce the fluctuation or to make it better? Am i having overfitting or under fitting problem here? I have tried using Dropout() but it just made it worse. What do i need to do to fix this problem?

Thanks before, Aquilla Setiawan Kanadi.

Answer 1

Firstly, the picture of the Validation Losses and Validation Accuracy is missing.

To answer your question, below might be the reason for your validation loss and validation accuracy fluctuations -

1. You have added approximately 1.25 times more weights to the base_model to build the model. (model Trainable Parameters 5115398 - base_model Trainable Parameters 2223872 = 2891526)

Program Statistics:

import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import GlobalAveragePooling2D, Dense
from keras.utils.layer_utils import count_params

class color:
   PURPLE = '\033[95m'
   CYAN = '\033[96m'
   DARKCYAN = '\033[36m'
   BLUE = '\033[94m'
   GREEN = '\033[92m'
   YELLOW = '\033[93m'
   RED = '\033[91m'
   BOLD = '\033[1m'
   UNDERLINE = '\033[4m'
   END = '\033[0m'

base_model = tf.keras.applications.MobileNetV2(weights='imagenet', include_top=False)

#base_model.summary()
trainable_count = count_params(base_model.trainable_weights)
non_trainable_count = count_params(base_model.non_trainable_weights)
print("\n",color.BOLD + '  base_model Statistics !' + color.END)
print("Trainable Parameters :", color.BOLD + str(trainable_count) + color.END)
print("Non Trainable Parameters :", non_trainable_count,"\n")

x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1025, activation='relu')(x)
x = Dense(1025, activation='relu')(x)
x = Dense(512, activation='relu')(x)
preds = Dense(3, activation='softmax')(x)
model = Model(inputs=base_model.input, outputs=preds)

#model.summary()
trainable_count = count_params(model.trainable_weights)
non_trainable_count = count_params(model.non_trainable_weights)
print(color.BOLD + '    model Statistics !' + color.END)
print("Trainable Parameters :", color.BOLD + str(trainable_count) + color.END)
print("Non Trainable Parameters :", non_trainable_count,"\n")

new_weights_added = count_params(model.trainable_weights) - count_params(base_model.trainable_weights)
print("Additional trainable weights added to the model excluding basel model trainable weights :", color.BOLD + str(new_weights_added) + color.END)

Output -

WARNING:tensorflow:`input_shape` is undefined or non-square, or `rows` is not in [96, 128, 160, 192, 224]. Weights for input shape (224, 224) will be loaded as the default.

   base_model Statistics !
Trainable Parameters : 2223872
Non Trainable Parameters : 34112 

    model Statistics !
Trainable Parameters : 5115398
Non Trainable Parameters : 34112 

Additional trainable weights added to the model excluding basel model trainable weights : 2891526

2. You are training complete model weights (MobileNetV2 weights and additional layers weights).

Solution to your problem is -

1. Customize the additional layers in such way that the new trainable parameters are minimum in comparison to base_model trainable parameters. May be add max pooling layers and fewer dense layers.

2. Freeze the base model by base_model.trainable = False and just train the new layers that you have added on top of the MobileNetV2 layers.

OR

Unfreeze the top layers of the base model(MobileNetV2 layers) and set the bottom layers to be un-trainable. You can do it as below, where we are freezing the model till 100th layer and remaining layers will be trainable -

# Let's take a look to see how many layers are in the base model
print("Number of layers in the base model: ", len(base_model.layers))

# Fine-tune from this layer onwards
fine_tune_at = 100

# Freeze all the layers before the `fine_tune_at` layer
for layer in base_model.layers[:fine_tune_at]:
  layer.trainable =  False

Output -

Number of layers in the base model:  155

3. Train the model with hyper parameter tuning. You can find more about hyper parameter tuning here.