Changing MobileNet Dropout After Loading

Question

I am working a transfer learning problem. When I create a new model from just the Mobilenet, I set a dropout.

base_model = MobileNet(weights='imagenet', include_top=False, input_shape=(200,200,3), dropout=.15)
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(10, activation='softmax')(x)

I save models as I train using model_checkpoint_callback. As I train I find where overfitting is happening and adjust the amount of frozen layers and the learning rate. Can I also adjust dropout when I save a loaded model again?

I saw this answer but there are no actual dropout layers in Mobilenet, so this

for layer in model.layers:
    if hasattr(layer, 'rate'):
        print(layer.name)
        layer.rate = 0.5

doesn't do anything.

Andrew Holmgren · Accepted Answer · 2020-07-07T22:59:46.760

In the past, you had to clone the model for the new dropout to take. I haven't tried it recently.

# This code allows you to change the dropout
# Load model from .json
model.load_weights(filenameToModelWeights) # Load weights
model.layers[-2].rate = 0.04  # layer[-2] is my dropout layer, rate is dropout attribute
model = keras.models.clone(model) # If I do not clone, the new rate is never used. Weights are re-init now.
model.load_weights(filenameToModelWeights) # Load weights
model.predict(x)

credit to

http://www.gergltd.com/home/2018/03/changing-dropout-on-the-fly-during-training-time-test-time-in-keras/

If the model doesn't have dropout layers to even begin with, as with Keras's pretrained mobilenet, you'll have to add them with methods. Here's one way you could do it.

For adding in a single layer

def insert_single_layer_in_keras(model, layer_name, new_layer):
    layers = [l for l in model.layers]

    x = layers[0].output
    for i in range(1, len(layers)):
        x = layers[i](x)
        # add layer afterward
        if layers[i].name == layer_name:
            x = new_layer(x)

    new_model = Model(inputs=layers[0].input, outputs=x)
    return new_model

For systematically adding a layer

def insert_layers_in_model(model, layer_common_name, new_layer):
    import re

    layers = [l for l in model.layers]
    x = layers[0].output
    layer_config = new_layer.get_config()
    base_name = layer_config['name']
    layer_class = type(dropout_layer)
    for i in range(1, len(layers)):
        x = layers[i](x)
        match = re.match(".+" + layer_common_name + "+", layers[i].name)
        # add layer afterward
        if match:
            layer_config['name'] = base_name + "_" + str(i)  # no duplicate names, could be done different
            layer_copy = layer_class.from_config(layer_config)
            x = layer_copy(x)

    new_model = Model(inputs=layers[0].input, outputs=x)
    return new_model

Run like this

import tensorflow as tf
from tensorflow.keras.applications.mobilenet import MobileNet
from tensorflow.keras.layers import Dropout
from tensorflow.keras.models import Model

base_model = MobileNet(weights='imagenet', include_top=False, input_shape=(192, 192, 3), dropout=.15)

dropout_layer = Dropout(0.5)
# add single layer after last dropout
mobile_net_with_dropout = insert_single_layer_in_model(base_model, "conv_pw_13_bn", dropout_layer)
# systematically add layers after any batchnorm layer
mobile_net_with_multi_dropout = insert_layers_in_model(base_model, "bn", dropout_layer)

By the way, you should absolutely experiment, but it's unlikely you want additional regularization on top of batchnorm for a small net like mobilenet.

As I mentioned,```if hasatrr(layer, 'rate')``` does not work, so there are no layers with ```rate``` as an attribute — theastronomist, Jul 07 '20 at 15:30
@theastronomist If the dropout layers aren't there to begin with you'll have to add them, see the edit for one way. — Andrew Holmgren, Jul 07 '20 at 23:00

Changing MobileNet Dropout After Loading

1 Answers1