How do I fit the model of two concatenate LSTM in keras?

Question

Hi I'm new in keras and I concatenate two LSTM in keras. The dataset is a univariate time series which was split by the method sliding window. Then I reshaped it as a tesor of [samples, features,time steps]. However when I tried to fit the model the following error appears.

TypeError: in user code:

/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
    return step_function(self, iterator)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py:796 step_function  **
    outputs = model.distribute_strategy.run(run_step, args=(data,))
/usr/local/lib/python3.8/dist-packages/tensorflow/python/distribute/distribute_lib.py:1211 run
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/distribute/distribute_lib.py:2585 call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/distribute/distribute_lib.py:2945 _call_for_each_replica
    return fn(*args, **kwargs)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py:789 run_step  **
    outputs = model.train_step(data)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py:747 train_step
    y_pred = self(x, training=True)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer.py:985 __call__
    outputs = call_fn(inputs, *args, **kwargs)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/sequential.py:386 call
    outputs = layer(inputs, **kwargs)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer.py:982 __call__
    self._maybe_build(inputs)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer.py:2643 _maybe_build
    self.build(input_shapes)  # pylint:disable=not-callable
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/utils/tf_utils.py:323 wrapper
    output_shape = fn(instance, input_shape)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/layers/merge.py:500 build
    del reduced_inputs_shapes[i][self.axis]

TypeError: list indices must be integers or slices, not ListWrapper

How should I pass trainX and testX? I want to do somenthing like in this image:

LSTMdata

The code is the following

 trainX = numpy.reshape(trainX, (trainX.shape[0],1, trainX.shape[1]))
 testX = numpy.reshape(testX, (testX.shape[0], 1,testX.shape[1]))

 model1= Sequential()
 model1.add(LSTM(6, input_shape=(1,look_back)))

 model2 = Sequential()
 model2.add(LSTM(6, input_shape=(1,look_back)))

 model = Sequential()
 model.add(Concatenate([model1, model1]))
 model.add(Dense(1))

 model.compile(loss='mean_squared_error', optimizer='adam', metrics = ['mse'])

 model.fit([trainX, trainX], trainY, epochs=50, batch_size=1, verbose=1)

What is the tensorflow version you are using? Do you use classes in your code before? — Florida Man, Nov 11 '20 at 20:17
Check this:https://stackoverflow.com/questions/52671481/why-are-variables-defined-with-self-automatically-given-a-listwrapper-while apparently similar issues are reported in Tensorflow 2.2.0 So it could be a Tensorflow version issue. — Florida Man, Nov 11 '20 at 20:19
Hi. thanks for your answer. I have the 2.3.1 version and I'm not using classes — Andres, Nov 12 '20 at 21:57
look at e.g., `trainX.shape[0]` separate before and check what type it is. It should be an int. If it is not an int, you can set it in by e.g. `x=int(trainX.shape[0])` — Florida Man, Nov 12 '20 at 22:59

MohammadHosseinZiyaaddini · Accepted Answer · 2020-11-13T10:41:32.870

You need to use Functional API in Keras.

Your model with Functional API:

from tensorflow.keras.layers import Dense, Concatenate, LSTM, Input, Flatten
from tensorflow.keras.models import Model

look_back = 3200  # just for running the code I used this number

# Model architecture 

inputs = Input(shape=(1,look_back),name='Input_1')

lstm1 = LSTM(6, name='LSTM_1')(inputs)

lstm2 = LSTM(6, name='LSTM_2')(inputs)

concatenated = Concatenate( name='Concatenate_1')([lstm1,lstm2])

output1 = Dense(1, name='Dense_1')(concatenated)

model = Model(inputs=inputs, outputs=output1)

Now let's see the architecture:

model.summary()

output:

Model: "functional_13"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
Input_1 (InputLayer)            [(None, 1, 3200)]    0                                            
__________________________________________________________________________________________________
LSTM_1 (LSTM)                   (None, 6)            76968       Input_1[0][0]                    
__________________________________________________________________________________________________
LSTM_2 (LSTM)                   (None, 6)            76968       Input_1[0][0]                    
__________________________________________________________________________________________________
Concatenate_1 (Concatenate)     (None, 12)           0           LSTM_1[0][0]                     
                                                                 LSTM_2[0][0]                     
__________________________________________________________________________________________________
Dense_1 (Dense)                 (None, 1)            13          Concatenate_1[0][0]              
==================================================================================================
Total params: 153,949
Trainable params: 153,949
Non-trainable params: 0
__________________________________________________________________________________________________

Also, we can take a better look at the layers and their connections by checking the model plot

The model plot:

from tensorflow.keras.utils import plot_model

plot_model(model)

output:

Now let's train the model:

I created some dummy data with sklearn for training the model and everything works fine.

Training the model:

from sklearn.datasets import make_blobs
train_x , train_y = make_blobs(n_samples=1000, centers=2, n_features=look_back,random_state=0)

train_x = train_x.reshape(train_x.shape[0], 1, train_x.shape[1])

model.compile(loss='mean_squared_error', optimizer='adam', metrics = ['mse'])
model.fit(train_x, train_y, epochs=5, batch_size=1, verbose=1)

output:

Epoch 1/5
1000/1000 [==============================] - 2s 2ms/step - loss: 0.0133 - mse: 0.0133
Epoch 2/5
1000/1000 [==============================] - 2s 2ms/step - loss: 1.4628e-13 - mse: 1.4628e-13
Epoch 3/5
1000/1000 [==============================] - 2s 2ms/step - loss: 2.2808e-14 - mse: 2.2808e-14
Epoch 4/5
1000/1000 [==============================] - 2s 2ms/step - loss: 5.2458e-15 - mse: 5.2458e-15
Epoch 5/5
1000/1000 [==============================] - 2s 2ms/step - loss: 1.1384e-15 - mse: 1.1384e-15
<tensorflow.python.keras.callbacks.History at 0x7f5fe4ce9f28>

Hi. Thanks a lot. It works !. There is some difference between creating this model using concatenate and the sequential model of two LSTM? What does the concatenation do? — Andres, Nov 15 '20 at 23:37
Hi. YW the [concatenate layer](https://keras.io/api/layers/merging_layers/concatenate/) is a layer that concatenates a list of inputs. — MohammadHosseinZiyaaddini, Nov 16 '20 at 04:42
And, the model is concatenating the hidden states of the LSTM layers by using the concatenate layer. — MohammadHosseinZiyaaddini, Nov 16 '20 at 04:51
I believe these links are also helpful: [lstm return sequences](https://machinelearningmastery.com/return-sequences-and-return-states-for-lstms-in-keras/), [different ways of building Keras models](https://www.pyimagesearch.com/2019/10/28/3-ways-to-create-a-keras-model-with-tensorflow-2-0-sequential-functional-and-model-subclassing/) — MohammadHosseinZiyaaddini, Nov 16 '20 at 04:57

Florida Man · Answer 2 · 2020-11-13T09:38:05.830

Try to write it like this:

...

model1= Sequential()
model1.add(LSTM(6, input_shape=(1,look_back)))

model2 = Sequential()
model2.add(LSTM(6, input_shape=(1,look_back)))

concatenated_models = concatenate([model1, model2])
out = Dense(1, activation='softmax', name='output')(concatenated_models)

added_model = Model([model1, model2], out)

added_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

added_model.fit([trainX, trainX], trainY, epochs=50, batch_size=1, verbose=1)

That might be working better. Not 100% sure (quite late here ;-))

How do I fit the model of two concatenate LSTM in keras?

2 Answers2