7

I am trying to implement a denoising autoencoder with an LSTM layer in between. The architecture goes following. 

FC layer -> FC layer -> LSTM cell -> FC layer -> FC layer.

I am unable to understand how my input dimension should be to implement this architecture?

I tried the following code

batch_size = 1
model = Sequential()
model.add(Dense(5, input_shape=(1,)))
model.add(Dense(10))
model.add(LSTM(32))
model.add(Dropout(0.3))
model.add(Dense(5))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(trainX, trainY, nb_epoch=100, batch_size=batch_size, verbose=2)

My trainX is [650,20,1] vector. It is a time series data in with only one feature.

I am getting following error

ValueError                                Traceback (most recent call last)
<ipython-input-20-1248a33f6518> in <module>()
      3 model.add(Dense(5, input_shape=(1,)))
      4 model.add(Dense(10))
----> 5 model.add(LSTM(32))
      6 model.add(Dropout(0.3))
      7 model.add(Dense(5))

/usr/local/lib/python2.7/dist-packages/keras/models.pyc in add(self, layer)
    330                  output_shapes=[self.outputs[0]._keras_shape])
    331         else:
--> 332             output_tensor = layer(self.outputs[0])
    333             if isinstance(output_tensor, list):
    334                 raise TypeError('All layers in a Sequential model '

/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in __call__(self, x, mask)
    527             # Raise exceptions in case the input is not compatible
    528             # with the input_spec specified in the layer constructor.
--> 529             self.assert_input_compatibility(x)
    530 
    531             # Collect input shapes to build layer.

/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in assert_input_compatibility(self, input)
    467                                          self.name + ': expected ndim=' +
    468                                          str(spec.ndim) + ', found ndim=' +
--> 469                                          str(K.ndim(x)))
    470             if spec.dtype is not None:
    471                 if K.dtype(x) != spec.dtype:

ValueError: Input 0 is incompatible with layer lstm_10: expected ndim=3, found ndim=2
Nilay Thakor
  • 375
  • 1
  • 3
  • 13

2 Answers2

8

The dense layer can take sequences as input and it will apply the same dense layer on every vector (last dimension). Example :

You have a 2D tensor input that represents a sequence (timesteps, dim_features), if you apply a dense layer to it with new_dim outputs, the tensor that you will have after the layer will be a new sequence (timesteps, new_dim)

If you have a 3D tensor (n_lines, n_words, embedding_dim) that can be a document, with n_lines lines, n_words words per lines and embedding_dim dimensions for each word, applying a dense layer to it with new_dim outputs will get you a new doc tensor (3D) with shape (n_lines, n_words, new_dim)

You can see here the dimensions input and output that you can feed and get with the Dense() layer.

Nassim Ben
  • 11,473
  • 1
  • 34
  • 52
  • how do you know if this was an OP intention? This is not straightforward from the question. – Marcin Możejko Mar 10 '17 at 11:09
  • 1
    I have ran some NLP on his question to predict it :) (just guessed, might not be it, but since there are not much information I guess he has a beginner level so even if it wasn't his question I think it will help him anyway :) if not, well this answer is a few bytes on a server, won't kill anywone). – Nassim Ben Mar 10 '17 at 12:38
  • I'm asking - because I thought about some sort of `Embedding` with `linear` transfer learning modifications. In this case your answer might be harmful. The insight you provided is interesting but actually might be completely misleading. – Marcin Możejko Mar 10 '17 at 12:48
  • You are right, feel free to edit with a warning :) otherwise I could just delete the answer. – Nassim Ben Mar 10 '17 at 12:49
  • I'd just leave it - but I would also wait for the OP answer in a comment to specify what he actually wanted to do :) – Marcin Możejko Mar 10 '17 at 12:51
  • loved the answer!! – shivam13juna Dec 27 '18 at 05:46
0

Although Nassim Ben already explained the background, as Google brought me here, I would like to mention the tensorflow.keras.Layers.Reshape layer.

I originally came from a "how to implement dropout"-point-of-view, but ran into the same problem.

Albeit the different Layer classes (may) come with their own dropout-options already embedded, I like to have my own, separate tensorflow.keras.Layers.Dropout squished in-between (for it helps my weak mind keeping track of them).

With

model.add(Reshape((1, Xtrain1.shape[1])))

now in turn squished in-between layers of the form Dropout (or Dense for that matter) and LSTM, at least I persuade myself, one has a solution tying together different layers with different requirements in terms of tensor dimension.

Sinistrum
  • 65
  • 8