4

I have a task to classify seeds depending on the defect. I have around 14k images in 7 classes (they are not equal size, some classes have more photos, some have less). I tried to train Inception V3 from scratch and I've got around 90% accuracy. Then I tried transfer learning using pre-trained model with ImageNet weights. I imported inception_v3 from applications without top fc layers, then added my own like in documentation. I ended with the following code:

# Setting dimensions
img_width = 454
img_height = 227

###########################
# PART 1 - Creating Model #
###########################

# Creating InceptionV3 model without Fully-Connected layers
base_model = InceptionV3(weights='imagenet', include_top=False, input_shape = (img_height, img_width, 3))

# Adding layers which will be fine-tunned
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
predictions = Dense(7, activation='softmax')(x)

# Creating final model
model = Model(inputs=base_model.input, outputs=predictions)

# Plotting model
plot_model(model, to_file='inceptionV3.png')

# Freezing Convolutional layers
for layer in base_model.layers:
    layer.trainable = False

# Summarizing layers
print(model.summary())

# Compiling the CNN
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])

##############################################
# PART 2 - Images Preproccessing and Fitting #
##############################################

# Fitting the CNN to the images

train_datagen = ImageDataGenerator(rescale = 1./255,
                                   rotation_range=30,
                                   width_shift_range=0.2,
                                   height_shift_range=0.2,
                                   shear_range = 0.2,
                                   zoom_range = 0.2,
                                   horizontal_flip = True,
                                   preprocessing_function=preprocess_input,)

valid_datagen = ImageDataGenerator(rescale = 1./255,
                                   preprocessing_function=preprocess_input,)

train_generator = train_datagen.flow_from_directory("dataset/training_set",
                                                    target_size=(img_height, img_width),
                                                    batch_size = 4,
                                                    class_mode = "categorical",
                                                    shuffle = True,
                                                    seed = 42)

valid_generator = valid_datagen.flow_from_directory("dataset/validation_set",
                                                    target_size=(img_height, img_width),
                                                    batch_size = 4,
                                                    class_mode = "categorical",
                                                    shuffle = True,
                                                    seed = 42)

STEP_SIZE_TRAIN = train_generator.n//train_generator.batch_size
STEP_SIZE_VALID = valid_generator.n//valid_generator.batch_size

# Save the model according to the conditions  
checkpoint = ModelCheckpoint("inception_v3_1.h5", monitor='val_acc', verbose=1, save_best_only=True, save_weights_only=False, mode='auto', period=1)
early = EarlyStopping(monitor='val_acc', min_delta=0, patience=10, verbose=1, mode='auto')

#Training the model
history = model.fit_generator(generator=train_generator,
                         steps_per_epoch=STEP_SIZE_TRAIN,
                         validation_data=valid_generator,
                         validation_steps=STEP_SIZE_VALID,
                         epochs=25,
                         callbacks = [checkpoint, early])

But I've got terrible results: 45% accuracy. I thought it should be better. I have some hypothesis what could go wrong:

  • I trained from scratch on scaled images (299x299) and on non-scaled while transfer-learning (227x454) and it failed something (or maybe I failed dimensions order).
  • While transfer-learning I used preprocessing_function=preprocess_input (found article on the web that it is extremely important, so I decided to add that).
  • Added rotation_range=30, width_shift_range=0.2, height_shift_range=0.2, and horizontal_flip = True while transfer learning to augment data even more.
  • Maybe Adam optimizer is a bad idea? Should I try RMSprop for example?
  • Should I fine-tune some conv layers with SGD with small learning rate too?

Or did I failed something else?

EDIT: I post a plot of training history. Maybe it contains valuable information:

History training plot

EDIT2: With changing parameters of InceptionV3:

InceptionV3 with changed parameters

VGG16 for comparison:

VGG16 for comparison

today
  • 32,602
  • 8
  • 95
  • 115
Michael S.
  • 155
  • 1
  • 11
  • What is the `preprocess_input` function you have used? – today Aug 23 '18 at 21:52
  • Oh, sorry. It is: from keras.applications.inception_v3 import InceptionV3, preprocess_input – Michael S. Aug 24 '18 at 09:19
  • Did you also use it when you trained inception from scratch? – today Aug 24 '18 at 09:21
  • No, I didn't. Now I am trying to train without it – Michael S. Aug 24 '18 at 10:12
  • Check Out [transfer_learning while BatchNorm](https://www.tensorflow.org/tutorials/images/transfer_learning#important_note_about_batchnormalization_layers) & [more on it](https://www.tensorflow.org/guide/keras/transfer_learning) –  Apr 12 '23 at 08:10

2 Answers2

5

@today, I found a problem. It is because of some changes in Batch Normalisation layers and its behavior while freezing them. Mr. Chollet gave a workaround, but I used a Keras fork made by datumbox, which solved my problem. The main problem is described here:

https://github.com/keras-team/keras/pull/9965

Now I get ~85% accuracy and am trying to raise it.

today
  • 32,602
  • 8
  • 95
  • 115
Michael S.
  • 155
  • 1
  • 11
  • 1
    Thanks for letting me know that. Actually I have previously came across this issue in the questions on SO, but I have completely forgotten that. – today Aug 31 '18 at 08:27
  • I also had the same problem and I tried the above Keras fork made by datumbox. In short, it worked ideally. This user addresses the issue in his blog in detail. http://blog.datumbox.com/the-batch-normalization-layer-of-keras-is-broken/ – Dane Lee Aug 15 '19 at 06:48
  • How do you apply the patch? – Readler Dec 02 '19 at 19:09
2

If you want to preprocess the input using the preprocess_input method from Keras, then remove the rescale=1./255 argument. Otherwise, keep the rescale argument and remove the preprocessing_function argument. Plus, try a lower learning rate like 1e-4 or 3e-5 or 1e-5 (the default learning rate of Adam optimizer is 1e-3) if loss does not decrease:

from keras.optimizers import Adam

model.compile(optimizer = Adam(lr=learning_rate), ...)

Edit: After adding the training plot, you can see that is it overfitting on the training set. You can:

  • add some kind of regularization like a Dropout layer,
  • or decrease the network size by lowering the number of units in the Dense layer which is before the last layer.
today
  • 32,602
  • 8
  • 95
  • 115
  • I will try it also. I will let you know about results – Michael S. Aug 24 '18 at 10:13
  • PS. I added a plot of training history. Do you think learning rate is too high? – Michael S. Aug 24 '18 at 10:24
  • @MichaelS. Oh! It shows overfitting. Yes, lowering the learning rate might help. But you need to add some kind of regularization. Adding a dropout layer might also help in this case. Or decrease the network size by decreasing the number of units in the Dense layer before last layer. 1024 might be too much! – today Aug 24 '18 at 10:38
  • Where should I add Dropout? Somewhere inside InceptionV3 net or just before fully-connected layers? Should I try 512 or even less units? – Michael S. Aug 24 '18 at 10:43
  • Also, try removing `GlobalAveragePooling2D` layer – Sreeram TP Aug 24 '18 at 10:44
  • @MichaelS. You can add it right after the pooling layer. I think 128 or 256 would work in this case, but I can't give you a definite answer. You must experiment, since it entirely depends on the data you have and the difficulty of the problem you are trying to solve. As it is often mentioned, deep learning is more of an art than a science. You must experiment, get feedback, change parameters based on the feedback and then again experiment... – today Aug 24 '18 at 10:48
  • I will try than. What about decay in optimizer? Should I try it? – Michael S. Aug 24 '18 at 10:49
  • @MichaelS. Yes of course you can (and should), but first just change the learning rate and see what you get. If you didn't get good results at all, change the optimizer to `RMSprop`. – today Aug 24 '18 at 10:52
  • @Sreeram TP - why is that? – Michael S. Aug 24 '18 at 10:52
  • Both Flatten and GlobalAveragePooling2D are valid options to use. So is GlobalMaxPooling2D. Flatten will result in a larger Dense layer afterwards, which is more expensive and may result in worse overfitting. But if you have lots of data, it might also perform better. As usual, it depends completely on your problem. So if you have a large dataset, try using flatten. Provided that you have made proper regularizations – Sreeram TP Aug 24 '18 at 11:37
  • @today, Unfortunately, changing learning rate, optimizer, adding droput and lowering units in Dense layer doesn't work. It gives only small changes, accuracy ~60%. I tried to transfer-learning VGG16 also. I got 95% accuracy without much problem. Is InceptionV3 so hard to train, am I doing something wrong or maybe the pretrained model is broken somehow? – Michael S. Aug 24 '18 at 21:20
  • @MichaelS. Unfortunately I have not worked with InceptionV3 much. I have worked with VGG networks and I have been able to get good results same as you. So I can't give you a specific answer. However, I just googled and found these two tutorials which use inception for transfer learning: [here](https://becominghuman.ai/transfer-learning-retraining-inception-v3-for-custom-image-classification-2820f653c557) and [here](https://sefiks.com/2017/12/10/transfer-learning-in-keras-using-inception-v3/). Note that I haven't read them so I don't know whether they are good or not, but I hope they help. – today Aug 25 '18 at 08:47
  • @MichaelS. If you succeeded in achieving a high accuracy, please share it with us and let us know what the problem was. – today Aug 25 '18 at 08:49
  • @today, I will give you know if I achieve something – Michael S. Aug 25 '18 at 10:54