Using Masking-layer with ConvLSTM2D-layer in Keras

Question

I am trying to mask missing data from a Convolutional LSTM layer using Keras (2.0.6) with TensorFlow backend (1.2.1):

import keras
from keras.models import Sequential
from keras.layers import Masking, ConvLSTM2D

n_timesteps = 10
n_width = 64
n_height = 64
n_channels = 1

model = Sequential()
model.add(Masking(mask_value = 0., input_shape = (n_timesteps, n_width, n_height, n_channels)))
model.add(ConvLSTM2D(filters = 64, kernel_size = (3, 3)))

However I am getting the following ValueError:

ValueError: Shape must be rank 4 but is rank 2 for 'conv_lst_m2d_1/while/Tile' (op: 'Tile') with input shapes: [?,64,64,1], [2].

How can I use Masking with the ConvLSTM2D-layer?

score 0 · Answer 1 · answered Jul 12 '18 at 15:49

It seems that Keras has problems when applying the Masking layer to arbitrary tensors such as images. I could find a (far from ideal) workaround that involves flattening the input before applying the mask layer. So considering the original model:

model = Sequential()
model.add(Masking(mask_value=0., input_shape=(n_timesteps, n_width, n_height, n_channels)))
model.add(ConvLSTM2D(filters=64, kernel_size=(3, 3)))

We can modify it as follows:

input_shape = (n_timesteps, n_width, n_height, n_channels)
model = Sequential()
model.add(TimeDistributed(Flatten(), input_shape=input_shape))
model.add(TimeDistributed(Masking(mask_value=0.)))
model.add(TimeDistributed(Reshape(input_shape[1:])))
model.add(ConvLSTM2D(filters=64, kernel_size=(3, 3)))

This solution adds extra computation in your graph but no additional parameters.

Using Masking-layer with ConvLSTM2D-layer in Keras

1 Answers1