Getting dimension error while passing the features extracted by CNN to LSTM

Question

I am doing video classification using deep learning in keras. I have extracted the features using VGG16 model whose shape is (7,7,512).I have around 55000 images. I passed this to LSTM layer but getting error of dimensions. Here is the code,

print len(train_data)
print train_data.shape[1:]
print train_data.shape
model = Sequential()
model.add(LSTM(128,input_shape=train_data.shape[1:]))
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(5, activation='softmax'))

Here is the output,

55936
(7, 7, 512)
(55936, 7, 7, 512)
Traceback (most recent call last):
File "train_rnn.py", line 135, in <module>
model.add(LSTM(128,input_shape=train_data.shape[1:]))
File "/usr/local/lib/python2.7/site-packages/keras/models.py", line 430, in add
layer(x)
File "/usr/local/lib/python2.7/site-    packages/keras/layers/recurrent.py", line 257, in __call__
return super(Recurrent, self).__call__(inputs, **kwargs)
File "/usr/local/lib/python2.7/site-packages/keras/engine/topology.py", line 534, in __call__
self.assert_input_compatibility(inputs)
File "/usr/local/lib/python2.7/site-packages/keras/engine/topology.py", line 433, in assert_input_compatibility
str(K.ndim(x)))
**ValueError**: Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=4`

Would you be able to share your code? I would like to know how you extracted the features, and then passing it to a LSTM. A demo of your script, perhaps? — Gautam J, Jun 01 '19 at 10:55

score 1 · Answer 1 · answered Apr 28 '17 at 15:01

Input shapes

3D tensor with shape (batch_size, timesteps, input_dim), (Optional) 2D tensors with shape (batch_size, output_dim).

Your input in 4D (including the input length).

try reshaping it to 3D:

train_data = train_data.reshape(train_data.shape[0],
                   train_data.shape[1] * train_data.shape[2],
                   train_data.shape[3])

Example (this will raise a valueError)

X = np.zeros((5, 7, 7, 512)

model = Sequential()
model.add(LSTM(128, input_shape=(7, 7, 512)))
model.add(Dense(1, activation='softmax')
model.compile(loss='binary_crossentropy', optimizer='sgd')

but this will not

X = np.zeros((5, 7, 7, 512)
X = X.reshape(5, 49, 512)

model = Sequential()
model.add(LSTM(128, input_shape=(49, 512)))
model.add(Dense(1, activation='softmax')
model.compile(loss='binary_crossentropy', optimizer='sgd')

How did you pass the extracted features into the LSTM? – TheJokerAEZ Jul 16 '19 at 19:14 — TheJokerAEZ, Jul 16 '19 at 19:14

Getting dimension error while passing the features extracted by CNN to LSTM

1 Answers1