Error when trying to output previous to last layer in Keras

Question

I am trying to output the previous to last Dense layer in a keras model. I first load the model architecture and the weights:

base_model = applications.ResNet50(weights = None,
                                            include_top = False, 
                                            input_shape = (image_size[0], image_size[1], nb_channels))
top_model = Sequential()
top_model.add(Flatten(input_shape=base_model.output_shape[1:]))
top_model.add(Dense(1024, init = 'glorot_uniform', activation='relu', name = 'last_layer_1024'))
top_model.add(Dropout(0.5))
top_model.add(Dense(nb_classes, activation = 'softmax', name = 'softmax_layer'))
top_model_tensor = top_model(base_model.output)
model = Model(inputs = base_model.input, outputs = top_model_tensor)
model.load_weights(weights_path)

Then I remove the last Dense layer by doing this:

model.layers[-1].pop()
#model.outputs = [model.layers[-1].layers[-1].output]
#model.layers[-1].layers[-1].outbound_nodes = []

If I uncomment the commented lines, I get this error: InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'flatten_1_input' with dtype float. If I keep them commented, the last dense layer is NOT effectively removed (by that I mean that when I call predict on model, I still get the output of the last dense layer). How can I solve this issue?

Also, if there is a different method to get the model to output the previous to last dense layer, I can take that as an answer too (instead of trying to fix this way of doing it).

Another solution that does not work is to just cut the long model after you load weights by simply doing this:

short_top_model = Model(top_model.input, top_model.get_layer('last_layer_1024').output)

You get the following error:

RuntimeError: Graph disconnected: cannot obtain value for tensor Tensor("flatten_1_input:0", shape=(?, 1, 1, 2048), dtype=float32, device=/device:GPU:2) at layer "flatten_1_input". The following previous layers were accessed without issue: []

Daniel Möller · Accepted Answer · 2017-11-30T13:14:13.207

2

Trying to cut models, change their inputs/outputs etc. sounds not what keras expects from users.

You should just create another model that follows the same path but ends earlier:

#do this "before" creating "top_model_tensor".
short_top = Model(
                  top_model.input,
                  top_model.get_layer('last_layer_1024').output
                 )

top_model_out = top_model(base_model.output)
short_top_out = short_top(base_model.output)

model = Model(base_model.input,top_model_out)
short_model = Model(base_model.input,short_top_out)

Choose which one to use depending on the expected results. Training one updates the other.

edited Nov 30 '17 at 13:14

answered Nov 30 '17 at 12:24

Daniel Möller

84,878
18
192
214

Hi Daniel, I phrased my question wrong. What I actually want to output is the 1024 Dense layer, not the nb_classes Dense layer. – Jadiel de Armas Nov 30 '17 at 12:48
Will I have trouble loading weights to a shorter model? Because the weights were trained in the longer model. – Jadiel de Armas Nov 30 '17 at 12:53
I will try now. – Jadiel de Armas Nov 30 '17 at 12:53
If you do it like this, you don't need to load weigths. The weights in both models are shared. – Daniel Möller Nov 30 '17 at 12:54
THe issue is that I already have weights that I trained a while ago. I trained them with the long model. But I now want to use them with the short model. – Jadiel de Armas Nov 30 '17 at 12:55
1

Load them in the long model. Save them from the short model later if you want. – Daniel Möller Nov 30 '17 at 12:56
Now get the following error: `RuntimeError: Graph disconnected: cannot obtain value for tensor Tensor("flatten_1_input:0", shape=(?, 1, 1, 2048), dtype=float32, device=/device:GPU:2) at layer "flatten_1_input". The following previous layers were accessed without issue: []` – Jadiel de Armas Nov 30 '17 at 13:02
Ok, this is tested now. – Daniel Möller Nov 30 '17 at 13:16
I have edited the original question to give good context to your answer. – Jadiel de Armas Nov 30 '17 at 13:27

score 0 · Answer 2 · answered Dec 07 '18 at 07:12

A shorter version of the above answer.

#again create connection between two model
feature_vec_model = Model(
                  top_model.input,
                  top_model.get_layer('last_layer_1024').output
                 )
feature_vec_model_output = feature_vec_model(base_model.output)
#Connection created


# Define final connected model & load pretrained weights
complete_feature_vec_model = Model(base_model.input,feature_vec_model_output)
complete_feature_vec_model.load_weights("path_to_model")

Error when trying to output previous to last layer in Keras

2 Answers2