10

I am trying to freeze the free trained VGG16's layers ('conv_base' below) and add new layers on top of them for feature extracting. I expect to get same prediction results from 'conv_base' before(ret1) / after(ret2) fit of model but it is not. Is this wrong way to check weight freezing?

loading VGG16 and set to untrainable

conv_base  = applications.VGG16(weights='imagenet', include_top=False, input_shape=[150, 150, 3]) 
conv_base.trainable = False

result before model fit

ret1 = conv_base.predict(np.ones([1, 150, 150, 3]))

add layers on top of the VGG16 and compile a model

model = models.Sequential()
model .add(conv_base)
model .add(layers.Flatten())
model .add(layers.Dense(10, activation='relu'))
model .add(layers.Dense(1, activation='sigmoid'))
m.compile('rmsprop', 'binary_crossentropy', ['accuracy'])

fit the model

m.fit_generator(train_generator, 100, validation_data=validation_generator, validation_steps=50)

result after model fit

ret2 = conv_base.predict(np.ones([1, 150, 150, 3]))

hope this is True but it is not.

np.equal(ret1, ret2)
Community
  • 1
  • 1
page
  • 121
  • 1
  • 4
  • Have you tried freezing all the layers? `for l in conv_base.layers: l.trainable=False` – Daniel Möller Nov 09 '17 at 14:50
  • If it doesn't work, maybe you need to set it as trainable after adding it to the sequential model... – Daniel Möller Nov 09 '17 at 14:51
  • @DanielMöller yes, that one works perfectly. but i don't get this why model.trainable works differently against the "layer.trainable for layer in model.layers". – page Nov 09 '17 at 16:00

3 Answers3

12

This is an interesting case. Why something like this happen is caused by the following thing:

You cannot freeze a whole model after compilation and it's not freezed if it's not compiled

If you set a flag model.trainable=False then while compiling keras sets all layers to be not trainable. If you set this flag after compilation - then it will not affect your model at all. The same - if you set this flag before compiling and then you'll reuse a part of a model for compiling another one - it will not affect your reused layers. So model.trainable=False works only when you'll apply it in a following order:

# model definition
model.trainable = False
model.compile()

In any other scenario it wouldn't work as expected.

Marcin Możejko
  • 39,542
  • 10
  • 109
  • 120
10

You must freeze layers individually (before compilation):

for l in conv_base.layers: 
    l.trainable=False

And if this doesn't work, you should probably use the new sequential model to freeze the layers.

If you have models in models you should do this recursively:

def freezeLayer(layer):
    layer.trainable = False
    if hasattr(layer, 'layers'):
        for l in layer.layers:
            freezeLayer(l)

freezeLayer(model)
Daniel Möller
  • 84,878
  • 18
  • 192
  • 214
  • 1
    This is the correct answer. `model.trainable=False` won't work, instead `for layer in model.layers: layer.trainable=False` worked for me – soulmachine Dec 01 '18 at 01:59
1

The top-rated answer does not work. As suggested by Keras official documentation (https://keras.io/getting-started/faq/), it should be performed per layer. Although there is a parameter "trainable" for a model, it is probably not implemented yet. The safest way is to do as follows:

for layer in model.layers:
    layer.trainable = False
model.compile()
Tristan_
  • 81
  • 1
  • 3