How to split a model trained in keras?

Question

I trained a model with 4 hidden layers and 2 dense layers and I have saved that model.

Now I want to load that model and want to split into two models, one with one hidden layers and another one with only dense layers.

I have splitted the model with hidden layer in the following way

model = load_model ("model.hdf5")
HL_model = Model(inputs=model.input, outputs=model.layers[7].output)

Here the model is loaded model, in the that 7th layer is my last hidden layer. I tried to split the dense in the like

DL_model = Model(inputs=model.layers[8].input, outputs=model.layers[-1].output)

and I am getting error

TypeError: Input layers to a `Model` must be `InputLayer` objects.

After splitting, the output of the HL_model will the input for the DL_model.

Can anyone help me to create a model with dense layer?

PS : I have tried below code too

from keras.layers import Input 
inputs = Input(shape=(9, 9, 32), tensor=model_1.layers[8].input)
model_3 = Model(inputs=inputs, outputs=model_1.layers[-1].output)

And getting error as

RuntimeError: Graph disconnected: cannot obtain value for tensor Tensor("conv2d_1_input:0", shape=(?, 144, 144, 3), dtype=float32) at layer "conv2d_1_input". The following previous layers were accessed without issue: []

here (144, 144, 3) in the input image size of the model.

jdehesa · Accepted Answer · 2018-03-09T13:20:02.400

13

You need to specify a new Input layer first, then stack the remaining layers over it:

DL_input = Input(model.layers[8].input_shape[1:])
DL_model = DL_input
for layer in model.layers[8:]:
    DL_model = layer(DL_model)
DL_model = Model(inputs=DL_input, outputs=DL_model)

edited Mar 09 '18 at 13:20

answered Mar 09 '18 at 12:58

jdehesa

58,456
7
77
121

Thanks, I followed the same as your suggested. I got the error `ValueError: The shape of the input to "Flatten" is not fully defined (got (None, 9, 9, 32). Make sure to pass a complete "input_shape" or "batch_input_shape" argument to the first layer in your model.` Then I changed `DL_input = Input(model.layers[8].input_shape)` to `DL_input = Input((9, 9, 32)` which is the actual shape of the layer. And now it is working. – Akhilesh Mar 09 '18 at 13:07
It was very informative so I have up-voted. Please do the required changes so that I can accept it as an answer. – Akhilesh Mar 09 '18 at 13:10
@Akhilesh Right, I've edited the first line (the shape passed to `Input` should not include the first batch dimension, but `input_shape` does), check if it works now. – jdehesa Mar 09 '18 at 13:14
2

does this method transport the weights and biases? – Arjun Jun 06 '19 at 05:10
this wont work if your network is not totally sequential. For example, if it has skip connections – Scratch Feb 25 '21 at 15:16
@Arjun yes, this keeps wights and biases and works like charm out of the box. – vedrano Feb 02 '23 at 18:55

score 0 · Answer 2 · answered Jun 18 '21 at 05:14

A little more generic. You can use the following function to split a model

from keras.layers import Input
from keras.models import Model
def get_bottom_top_model(model, layer_name):
    layer = model.get_layer(layer_name)
    bottom_input = Input(model.input_shape[1:])
    bottom_output = bottom_input
    top_input = Input(layer.output_shape[1:])
    top_output = top_input

    bottom = True
    for layer in model.layers:
        if bottom:
            bottom_output = layer(bottom_output)
        else:
            top_output = layer(top_output)
        if layer.name == layer_name:
            bottom = False

    bottom_model = Model(bottom_input, bottom_output)
    top_model = Model(top_input, top_output)

    return bottom_model, top_model
bottom_model, top_model = get_bottom_top_model(model, "dense_1")

Layer_name is just the name of the layer that you want to split at.

If I use that, I get `TypeError: Dimension value must be integer or None or have an __index__ method, got value '(None, 224, 224, 3)' with type ''` here `bottom_input = Input(model.input_shape[1:])` — pceccon, May 11 '22 at 16:56

score 0 · Answer 3 · answered Apr 24 '23 at 10:17

For sequential keras models, we can do the following:

Suppose, this is my model:

hidden_units = 256
dropout = 0.45
model = Sequential()
model.add(Dense(hidden_units, activation='relu', input_dim=784))
model.add(Dropout(dropout))
model.add(Dense(hidden_units, activation='relu'))
model.add(Dropout(dropout))
model.add(Dense(hidden_units, activation='relu'))
model.add(Dropout(dropout))
model.add(Dense(10, activation='softmax'))
model.summary()

And I want to split it at the third layer (i.e., dense_9)

Model: "sequential_2"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_8 (Dense)             (None, 256)               200960    
                                                                 
 dropout_6 (Dropout)         (None, 256)               0         
                                                                 
 dense_9 (Dense)             (None, 256)               65792     
                                                                 
 dropout_7 (Dropout)         (None, 256)               0         
                                                                 
 dense_10 (Dense)            (None, 256)               65792     
                                                                 
 dropout_8 (Dropout)         (None, 256)               0         
                                                                 
 dense_11 (Dense)            (None, 10)                2570      
                                                                 
=================================================================
Total params: 335,114
Trainable params: 335,114
Non-trainable params: 0
_________________________________________________________________

The tail of the model, i.e., the model that takes the user input:

model_tail = keras.Sequential(model.layers[:3])

And the head is just composed of layers after the third layer. However, we need to add an input layer to it. This Input layer need to consume the output of the tail model. (also mentioned in another answer)

input_tail =  keras.layers.Input(model.layers[3].input_shape[1:])
model_head = keras.Sequential([input_tail] + model.layers[3:])

And we use it as follows:

tail_pred = model_tail.predict(some_input)
head_pred = model_head.predict(tail_pred)
head_pred.argmax(axis=-1) # assuming a simple classifier

How to split a model trained in keras?

3 Answers3