validation accuracy is not increasing training ResNet50

Question

I was doing fine-tuning with ResNet50 model for face recognition using data agumentation, but observed that the model accuracy was increasing but validation accuracy from the very starting point is not imporving, i am not getting where is it getting wrong, please review my code.

I have tried manipulating the top layers which i have added but it didn't helped.

import os
os.environ['KERAS_BACKEND'] = 'tensorflow'
from keras.applications import ResNet50
from keras.models import Sequential
from keras.layers import Dense, Flatten, GlobalAveragePooling2D,Input,Dropout

num_classes = 13

base = ResNet50(include_top=False, weights='resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5',input_tensor=Input(shape=(100,100,3)))
from keras.models import Model

x = base.output

#x = GlobalAveragePooling2D()(x)

x = Flatten()(x)

x = Dense(1024, activation = 'relu')(x)

x = Dropout(0.5)(x)

predictions = Dense(13, activation='softmax')(x)

model = Model(inputs=base.input, outputs=predictions)

for layers in base.layers:
    layers.trainable= False

model.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['accuracy'])
from keras.preprocessing.image import ImageDataGenerator

train_generator = ImageDataGenerator(featurewise_center=True,
                                rotation_range=20,
                                rescale=1./255,
                                shear_range=0.2,
                                zoom_range=0.2,
                                width_shift_range=0.2,
                                height_shift_range=0.2,
                                horizontal_flip=True)

test_generator = ImageDataGenerator(rescale=1./255)

from sklearn.model_selection import train_test_split

x_train,x_test,y_train,y_test = train_test_split(image,label,test_size=0.2,shuffle=True,random_state=0)

train_generator.fit(x_train)
test_generator.fit(x_test)

model.fit_generator(train_generator.flow(x_train,y_train,batch_size=32),
                       steps_per_epoch =10,epochs=50, 
                    validation_data=test_generator.flow(x_test,y_test))

Output :

Epoch 19/50
10/10 [==============================] - 105s 10s/step - loss: 1.9387 - acc: 0.3803 - val_loss: 2.6820 - val_acc: 0.0709
Epoch 20/50
10/10 [==============================] - 107s 11s/step - loss: 2.0725 - acc: 0.3230 - val_loss: 2.6689 - val_acc: 0.0709
Epoch 21/50
10/10 [==============================] - 103s 10s/step - loss: 1.8884 - acc: 0.3375 - val_loss: 2.6677 - val_acc: 0.0709
Epoch 22/50
10/10 [==============================] - 95s 10s/step - loss: 1.8265 - acc: 0.4051 - val_loss: 2.6799 - val_acc: 0.0709
Epoch 23/50
10/10 [==============================] - 100s 10s/step - loss: 1.8346 - acc: 0.3812 - val_loss: 2.6929 - val_acc: 0.0709
Epoch 24/50
10/10 [==============================] - 102s 10s/step - loss: 1.9547 - acc: 0.3352 - val_loss: 2.6952 - val_acc: 0.0709
Epoch 25/50
10/10 [==============================] - 104s 10s/step - loss: 1.9472 - acc: 0.3281 - val_loss: 2.7168 - val_acc: 0.0709
Epoch 26/50
10/10 [==============================] - 103s 10s/step - loss: 1.8818 - acc: 0.4063 - val_loss: 2.7071 - val_acc: 0.0709
Epoch 27/50
10/10 [==============================] - 106s 11s/step - loss: 1.8053 - acc: 0.4000 - val_loss: 2.7059 - val_acc: 0.0709
Epoch 28/50
10/10 [==============================] - 104s 10s/step - loss: 1.9601 - acc: 0.3493 - val_loss: 2.7104 - val_acc: 0.0709

Akash · Accepted Answer · 2018-10-28T23:08:47.450

3

It was happening because i was just directly adding the fully-connected layers without training it first, as mentioned in Blog of keras, https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

in order to perform fine-tuning, all layers should start with properly trained weights: for instance you should not slap a randomly initialized fully-connected network on top of a pre-trained convolutional base. This is because the large gradient updates triggered by the randomly initialized weights would wreck the learned weights in the convolutional base. In our case this is why we first train the top-level classifier, and only then start fine-tuning convolutional weights alongside it.

so answer is first train the top-model separately, then create a new model having ResNet50 model with its weight, with top-model and its weights on top of resnet model(base-model), then train it first by freezing the base-model(ResNet50) and the with the last layer of the base-model.

edited Oct 28 '18 at 23:08

answered Oct 28 '18 at 22:56

Akash

139
1
10

1

Could you post an example please ? – Anas Hadhri Dec 29 '19 at 13:49
yes could you provide your code? I have the same problem but don't know how to train the top layer seperately. – NECben067 Aug 11 '20 at 05:25
Hi @NECben067, currently i am not able to post the code as its been a long time since i did that, but i can make it understand from the example above. So the idea is to train a simple sequential model having 1 dense layer (as per above example, it totally depends upon you) on the data set and then add that model on the top of ResNet50 base model. The reason for doing it is stated above. – Akash Aug 12 '20 at 08:49
@NECben067 New link https://keras.io/guides/transfer_learning/ has a detailed description of how to train the top classifier along with complete code – shantanu pathak Feb 16 '21 at 14:06

validation accuracy is not increasing training ResNet50

1 Answers1

Linked