3

I'm trying to build a CNN + RNN using a pre-trained model from tensorflow-hub:

base_model = hub.KerasLayer('https://tfhub.dev/google/imagenet/resnet_v2_50/feature_vector/4', input_shape=(244, 244, 3)
base_model.trainable = False

model = Sequential()
model.add(TimeDistributed(base_model, input_shape=(15, 244, 244, 3)))
model.add(LSTM(512))
model.add(Dense(256, activation='relu'))
model.add(Dense(3, activation='softmax'))

adam = Adam(learning_rate=learning_rate)
model.compile(loss='categorical_crossentropy' , optimizer=adam , metrics=['accuracy'])
model.summary()

and this what I get:

2020-01-29 16:1

6:37.585888: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2494000000 Hz
2020-01-29 16:16:37.586205: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3b553f0 executing computations on platform Host. Devices:
2020-01-29 16:16:37.586231: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
Traceback (most recent call last):
  File "./RNN.py", line 45, in <module>
    model.add(TimeDistributed(base_model, input_shape=(None, 244, 244, 3)))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/base.py", line 457, in _method_wrapper
    result = method(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/sequential.py", line 178, in add
    layer(x)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 842, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/wrappers.py", line 256, in call
    output_shape = self.compute_output_shape(input_shape).as_list()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/wrappers.py", line 210, in compute_output_shape
    child_output_shape = self.layer.compute_output_shape(child_input_shape)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 639, in compute_output_shape
    raise NotImplementedError
NotImplementedError

any suggestions? Is it possible to convert a KerasLayer to Conv2D,... layers?

NickF_93
  • 479
  • 4
  • 16

3 Answers3

2

It doesn't seem like you can use the TimeDistributed layer for this problem. However, as you don't want Resnet to train and just need the output, you can do the following to avoid the TimeDistributed layer.

Instead of model.add(TimeDistributed(base_model, input_shape=(15, 244, 244, 3))), do

Option 1

# 2048 is the output size
model.add(
    Lambda(
        lambda x: tf.reshape(base_model(tf.reshape(x, [-1, 244, 244,3])),[-1, 15, 2048])
    , input_shape=(15, 244, 244, 3))
)

Option 2

If you don't want to depend too much on the output shape (this sacrifices performance though).

model.add(
    Lambda(
        lambda x: tf.stack([base_model(xx) for xx in tf.unstack(x, axis=1) ], axis=1)
    , input_shape=(15, 244, 244, 3))
)
thushv89
  • 10,865
  • 1
  • 26
  • 39
  • I think TimeDistributed and LSTM layers are nedded because I have sequences of 15 frames to classify. I noticed that if I use KerasLayer for a pretrained model from Tensorflow Hub it give me that error, if I define a ResNet50 from tf.keras.applications.ResNet50 it don't give me any error. What is the difference? I don't understand – NickF_93 Jan 30 '20 at 00:34
  • 1
    Your `KerasLayer` doesn't need to go through a `TimeDistributed` layer. All it's doing is giving a feature vector for each image and it's not optimized (the layer is frozen). It doesn't matter if you process the data with or without a timestep for the Resnet part. – thushv89 Jan 30 '20 at 00:51
  • 1
    And for the different `tf_hub` is different from `tf.keras.applications`. Models in `tf.keras.applications` are well integrated to `tf.keras` operations. Whereas `tf_hub` might not. – thushv89 Jan 30 '20 at 00:53
  • Ah ok, thanks for explanation. So it is not possible tu use a pre trained network for transfering learning to a CNN+RNN convolutional layer? – NickF_93 Jan 30 '20 at 08:38
  • 1
    @NickF_93 Unfortunately I don't see a way. I got the same error as you and couldn't get around it. – thushv89 Jan 30 '20 at 22:10
  • Ok, thanks. I also tried to use `tf.keras.applications` but I have the problem described here https://stackoverflow.com/questions/58607878/tensorflow-hub-vs-keras-application-performance-drop and here https://stackoverflow.com/questions/59994205/keraslayer-vs-tf-keras-applications-performances . – NickF_93 Jan 31 '20 at 07:52
0

See my answer here. The error is thrown because it can't compute the output shape, you might be able to solve your problem by implementing compute_output_shape manually.

Josef
  • 2,869
  • 2
  • 22
  • 23
0

If someone faces this issue, as @josef mentioned, the problem is that compute_output_shape is not implemented. You can overcome the problem by specifying the output shape of the layer:

extractor = tfhub.KerasLayer("https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/feature_vector/4",
                              input_shape=(IMG_SIZE, IMG_SIZE, CHANNELS),
                              output_shape=(EXTRACTOR_SIZE),
                              trainable=False)
    
model.add(keras.layers.Lambda(lambda x: extractor(x)))

As you can see, I also had to wrap the layer inside of a Lambda function since it looks like you can't wrap KerasLayer directly inside of a TimeDistributed layer.

In the code, EXTRACTOR_SIZE is 1280 but that is specific of MobileNet.

This workaround worked for me.

Facundo Santiago
  • 41
  • 1
  • 7