Encoder returning same states for every input Keras seq2seq

Question

I am using an Encoder Decoder seq2seq architecture in Keras, I'm passing a one-hot array of shape (num_samples, max_sentence_length, max_words) for training, and using teacher forcing.

#Encoder
latent_dim = 256
encoder_inputs = Input(shape=(None, max_words))
encoder = LSTM(latent_dim, return_state = True)
encoder_outputs, state_h, state_c = encoder(encoder_inputs)
encoder_states = [state_h, state_c]
#Decoder
decoder_inputs = Input(shape=(None, max_words))
decoder_lstm = LSTM(latent_dim, return_state = True, return_sequences = 
True)
decoder_outputs, _, _ = decoder_lstm(decoder_inputs, initial_state= 
encoder_states)
decoder_dense = Dense(max_words, activation = 'softmax')
decoder_outputs = decoder_dense(decoder_outputs)

For inference model:

# Inference model
encoder_model = Model(encoder_inputs, encoder_states)

decoder_state_input_h = Input(shape=(latent_dim,))
decoder_state_input_c = Input(shape=(latent_dim,))
decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]
decoder_outputs, state_h, state_c = decoder_lstm(
    decoder_inputs, initial_state=decoder_states_inputs)
decoder_states = [state_h, state_c]
decoder_outputs = decoder_dense(decoder_outputs)
decoder_model = Model(
    [decoder_inputs] + decoder_states_inputs,
    [decoder_outputs] + decoder_states)

I tried printing out the encoder_model states, but it always returns the same states for any input. Any help would be appreciated!

Answer 1

This is pretty much just from the Keras example, correct?

https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html

Did you train the model? Everything you have posted is the same as in the Keras documentation, so I don't think that could be the problem.

Here is an example that works, which is also based on the Keras documentation and looks pretty similar to what you have. Maybe try running through this and seeing where you are different?

https://github.com/JEddy92/TimeSeries_Seq2Seq/blob/master/notebooks/TS_Seq2Seq_Intro.ipynb