1

I have a LSTM feed-forward neural network as written below. For some reasons, I need to add a backward (recurrent) connection from layer3 to layer1 as well, which results in a loop in my model's architecture. How can I modify the below code to make such a loop in my model's architecture?

import tensorflow as tf
from tensorflow.keras.layers import Input, Dense, LSTM
from tensorflow.keras.models import Model

input_shape = ...  # Shape of your input data
hidden_units1 = ...  # Number of units in the first hidden layer
hidden_units2 = ...  # Number of units in the second hidden layer
hidden_units3 = ...  # Number of units in the third hidden layer
output_units = ...  # Number of output units

# Define the input layer
input_layer = Input(shape=(input_shape,))
layer1 = LSTM(units=hidden_units1, activation='relu', return_sequences=True)(input_layer)
layer2 = LSTM(units=hidden_units2, activation='relu', return_sequences=True)(layer1)
layer3 = LSTM(units=hidden_units3, activation='relu')(layer2)
output_layer = Dense(units=output_units, activation='softmax')(layer3)
model = Model(inputs=input_layer, outputs=output_layer)
Mohammad
  • 163
  • 2
  • 8
  • 1
    Should layer1's input on timestep x be the input layer on timestep x and the output of layer3 on timestep x-1? – mhenning Aug 01 '23 at 07:36
  • No, the final input of layer one is coming from its previous layer concatenated with the output of layer 3. – Mohammad Aug 01 '23 at 15:51
  • 1
    Well then it sounds like a hen-egg problem for me. For layer3 you need the output of layer1, and for layer1 you would need the output of layer3.. you can't get both if layer1 should depend on layer3. – mhenning Aug 01 '23 at 16:05
  • According to my search so far, it can be implemented as an iterative-neural network using a for loop with a certain number of iterations. – Mohammad Aug 01 '23 at 16:08
  • 1
    Please explain to me in other words how `layer1` can have `layer3` as input, if `layer3`'s input depends on `layer1`'s output? You need to compute one to compute the other, and then you already computed one. – mhenning Aug 02 '23 at 07:47
  • You can initialize 'layer3' first and start a for loop which makes an iterative neural network. – Mohammad Aug 02 '23 at 16:25
  • Did you mean that you will use the output of the previous(t-1 step) activation of layer3 as input to the current (t step)layer1 – Tou You Aug 02 '23 at 17:21
  • You can initialize the weights and bias for a layer, but you can't initialize its output. For an output of a layer you need an input, and then you have the dependencies on earlier layers again.. – mhenning Aug 03 '23 at 07:23

0 Answers0