Combine to Keras functional models

Question

I am trying to mimic this keras blog about fine tuning image classifiers. I would like to use the Inceptionv3 found on a fchollet repo.

Inception is a Model (functional API), so I can't just do model.add(top_model) which is reserved for Sequential.

How can I add combine two functional Models? Let's say I have

inputs = Input(shape=input_shape)
x = Flatten()(inputs)
predictions = Dense(4, name='final1')(x)

model1 = Model(input=inputs, output=predictions)

for the first model and

inputs_2 = Input(shape=(4,))
y = Dense(5)(l_inputs)
y = Dense(2, name='final2')(y)
predictions_2 = Dense(29)(y)
model2 = Model(input=inputs2, output=predictions2)

for the second. I now want an end-to-end that goes from inputs to predicions_2 and links predictions to inputs_2.

I tried using model1.get_layer('final1').output but I had a mismatch with types and I couldn't make it work.

Answer 1

I haven't tried this but according to the documentation functional models are callable, so you can do something like:

y = model2(model1(x))

where x is the data that goes to inputs and y is the result of predictions_2

Answer 2

I ran into this problem as well while fine tuning VGG16. Here's what worked for me and I imagine a similar approach can be taken for Inception V3. Tested on Keras 2.0.5 with Tensorflow 1.2 backend.

# NOTE: define the following variables
#       top_model_weights_path
#       num_classes
#       dense_layer_1 = 4096
#       dense_layer_2 = 4096

vgg16 = applications.VGG16(
    include_top=False,
    weights='imagenet',
    input_shape=(224, 224, 3))

# Inspect the model
vgg16.summary()

# This shape has to match the last layer in VGG16 (without top)
dense_input  = Input(shape=(7, 7, 512))
dense_output = Flatten(name='flatten')(dense_input)
dense_output = Dense(dense_layer_1, activation='relu', name='fc1')(dense_output)
dense_output = Dense(dense_layer_2, activation='relu', name='fc2')(dense_output)
dense_output = Dense(num_classes, activation='softmax', name='predictions')(dense_output)

top_model = Model(inputs=dense_input, outputs=dense_output, name='top_model')

# from: https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html
# note that it is necessary to start with a fully-trained
# classifier, including the top classifier,
# in order to successfully do fine-tuning
top_model.load_weights(top_model_weights_path)

block5_pool = vgg16.get_layer('block5_pool').output

# Now combine the two models
full_output = top_model(block5_pool)
full_model  = Model(inputs=vgg16.input, outputs=full_output)

# set the first 15 layers (up to the last conv block)
# to non-trainable (weights will not be updated)
# WARNING: this may not be applicable for Inception V3
for layer in full_model.layers[:15]:
    layer.trainable = False

# Verify things look as expected
full_model.summary()

# compile the model with a SGD/momentum optimizer
# and a very slow learning rate.
full_model.compile(
    loss='binary_crossentropy',
    optimizer=optimizers.SGD(lr=5e-5, momentum=0.9),
    metrics=['accuracy'])

# Train the model...

Answer 3

I think there are 2 options depending on what you need:

(a) predictions_1 and predictions_2 matter for you. In this case, you can train a network with 2 outputs. Here an example derived from your post:

input_shape = [3, 20]
inputs = Input(shape=input_shape)
x = Flatten()(inputs)
predictions_1 = Dense(4, name='predictions_1')(x)

# here the predictions_1 just corresponds to your next layer's input
y = Dense(5)(predictions_1)
y = Dense(2)(y)
predictions_2 = Dense(29, name='predictions_2')(y)

# you specify here that you have 2 outputs
model = Model(input=inputs, output=[predictions_1, predictions_2])

For the .fit and .predict, you can find a lot of details in https://keras.io/getting-started/functional-api-guide/, section: Multi-input and multi-output models.

(b) you are only interested in predictions_2. In this case, you can just do:

input_shape = [3, 20]
inputs = Input(shape=input_shape)
x = Flatten()(inputs)
predictions_1 = Dense(4, name='predictions_1')(x)

# here the predictions_1 just corresponds to your next layer's input
y = Dense(5)(predictions_1)
y = Dense(2)(y)
predictions_2 = Dense(29, name='predictions_2')(y)

# you specify here that your only output is predictions_2
model = Model(input=inputs, output=predictions_2)

Now as regards inception_v3. You can define by yourself the architecture and modify the deep layers inside according to your needs (giving to these layers specific names in order to avoid keras naming them automatically).

After that, compile your model and loads weights (as in https://keras.io/models/about-keras-models/ see function load_weights(..., by_name=True))

# you can load weights for only the part that corresponds to the true
# inception_v3 architecture. The other part will be initialized
# randomly
model.load_weights("inception_v3.hdf5", by_name=True)

This should solve your problem. By the way, you can find extra information here: https://www.gradientzoo.com. The doc. explains several saving / loading / fine-tuning routines ;)

Update: if you do not want to redefine your model from scratch you can do the following:

input_shape = [3, 20]

# define model1 and model2 as you want
inputs1 = Input(shape=input_shape)
x = Flatten()(inputs1)
predictions_1 = Dense(4, name='predictions_1')(x)
model1 = Model(input=inputs1, output=predictions_1)

inputs2 = Input(shape=(4,))
y = Dense(5)(inputs2)
y = Dense(2)(y)
predictions_2 = Dense(29, name='predictions_2')(y)
model2 = Model(input=inputs2, output=predictions_2)

# then define functions returning the image of an input through model1 or model2
def give_model1():
    def f(x):
        return model1(x)
    return f

def give_model2():
    def g(x):
        return model2(x)
    return g

# now you can create a global model as follows:
inputs = Input(shape=input_shape)
x = model1(inputs)
predictions = model2(x)
model = Model(input=inputs, output=predictions)

Answer 4

Drawing from filitchp's answer above, assuming the output dimensions of model1 match the input dimensions of model2, this worked for me:

model12 = Model(inputs=inputs, outputs=model2(model1.output))