Keras LSTM autoencoder with embedding layer

Question

I am trying to build a text LSTM autoencoder in Keras. I want to use an embedding layer but I'am not sure how to implement this. The code looks like this.

inputs = Input(shape=(timesteps, input_dim))
embedding_layer = Embedding(numfeats + 1,
                            EMBEDDING_DIM,
                            weights=[data_gen.get_embedding_matrix()],
                            input_length=maxlen,
                            trainable=False)

embedded_sequence = embedding_layer(inputs)
encoded = LSTM(num_units)(inputs)

decoded = RepeatVector(timesteps)(encoded)
decoded = LSTM(???, return_sequences=True)(decoded)

sequence_autoencoder = Model(inputs, decoded)

sequence_autoencoder.compile(loss='binary_crossentropy', optimizer='adam')

I am not sure how to decode the output into the target sequence (which is obviously the input sequence).

Did you figure out how to do this? – sariii Apr 06 '19 at 23:21 — sariii, Apr 06 '19 at 23:21

score 1 · Answer 1 · answered Nov 03 '19 at 18:07

1

You can first convert the word to embeddings and pass them to the fit()

expected_output = np.array([[embedding_matrix[word_index] for word_index in encoded_sequence] for encoded_sequence in padded_sequences])
history = lstm_autoencoder.fit(padded_sequences, expected_output, epochs=15, verbose=1)

answered Nov 03 '19 at 18:07

sebdei

11
1

My understanding is that this is an example of the first solution suggested by @ChrisYao. The expected output is the embedding of the input sequence and the output layer is a dense layer with shape (n_words, embedding_dim). The only issue is that it can be hard optimising this objective. – Alex_Gidiotis Nov 04 '19 at 11:36

score 0 · Accepted Answer · answered Jan 11 '18 at 21:29

0

There is no way to implement an inversed embedding layer in the decoder, because the embedding layer is not differentiable. There are probably other ways to work around:

construct the autoencoder from the output of the embedding layer, to a layer with a similar dimension. Then use the nearest neighbor or other algorithms to generate the word sequence from there.
construct an asymmetric autoencoder, using the time distributed layer and dense layers to reduce the dimension of LSTM output.

Hopefully this helps.

answered Jan 11 '18 at 21:29

Chris Yao

240
3
13

Can you please reference an example? – sariii Apr 06 '19 at 23:22
The first solution can work without needing the mapping into word tokens. The expected output is the embedding of the input sequence and the output layer is a dense layer with shape (n_words, embedding_dim). The only issue is that it can be hard optimising this objective. – Alex_Gidiotis Nov 04 '19 at 11:42

Keras LSTM autoencoder with embedding layer

2 Answers2

Linked