Why does Keras not return the full sequence of cell state in lstm layer?

Question

I am trying to implement an attention mechanism where I need the full sequence of the cell state (just like the full sequence of the hidden state). Keras LSTM however only returns the last cell state:

output, state_h, state_c = layers.LSTM(units=45, return_state=True, return_sequences=True)

state_c has shape (batch size, 1, 45) where output (which is the full sequence hidden state) has shape (batch size, 5, 45). 5 is the time window length

Why does Keras not return the full sequence cell state? and is there a better approach to get the full sequence of cell state than the approach below?

full_hidden, full_cell, outputs = [], [], []
state = None
input = layers.Input(shape=(time_window,features), dtype='float32')
output = layers.LSTM(units=45, return_state=True)

for i in range(time_window):
    input_t = input[:, i, :]
    input_t = tf.expand_dims(input_t, 1)
    out, state_h, state_c = lstm(input_t, initial_state=state)
    state = state_h, state_c
    full_hidden.append(state_h)
    full_cell.append(state_c)
    outputs.append(out)

What type of attention are you implementing? For some, you only need the last cell state. — TheExplodingGradient, Sep 22 '20 at 15:40
I am implementing [DA-RNN](https://arxiv.org/pdf/1704.02971.pdf). Equation 8 suggests I need every cell state for each time step. — bcsta, Sep 22 '20 at 15:46

score -1 · Answer 1 · answered Sep 24 '20 at 08:53

-1

You need to set the flag return_sequences to True to get all the temporal states. The flag return_state=True that you use makes the layer to return the final state.

answered Sep 24 '20 at 08:53

Jindřich

10,270
2
23
44

1

return_state=True still will not return the cell state for each time step. It returns the cell state at the last time step – bcsta Sep 25 '20 at 07:02
Yes, this is what I am saying, you need to set `return_sequences` to `True`. – Jindřich Sep 25 '20 at 07:11
2

again, that still return the cell state's last value – bcsta Sep 25 '20 at 15:17

Why does Keras not return the full sequence of cell state in lstm layer?

1 Answers1