How to reshape to make a 3d input for LSTM?

Question

I have 1002 times of observing the 10 features. I have concatenated together for doing preprocessing on the data and now, I need to reshape it to a 3D input data to be used in LSTM. I do not know using pd.df.groupby(['timestep']) is meaningful but I have used a tiny "np.reshape" function but seems does not determine and I got the error. The code is below:

train_dataset = dataset.sample(frac=0.8,random_state=0)
test_dataset = dataset.drop(train_dataset.index)

scaler = MinMaxScaler()
train_x = scaler.fit_transform(train_dataset)
test_x  = scaler.fit_transform(test_dataset)

data_train = train_x.reshape(1002, 1001, 11)
def build_model():
    model = tf.keras.Sequential([
        layers.LSTM(128,activation=tf.nn.relu,input_shape=train_dataset.shape[1:],return_sequences=True, return_state=True),
        layers.LSTM(128,activation=tf.nn.relu,return_sequences=True, return_state=True),
        layers.Dense(1)
    ])
    opt = tf.keras.optimizers.Adam(learning_rate=0.001)
    model.compile(loss='mean_squared_error', optimizer=opt, metrics= ['mean_squared_error','mean_absolute_error'])
    
    return model

and the error that I faced for the Dense layer is below:

Input 0 of layer lstm_10 is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: [None, 11]

Does this answer your question? [Keras : How should I prepare input data for RNN?](https://stackoverflow.com/questions/36992855/keras-how-should-i-prepare-input-data-for-rnn) — Marco Cerliani, Sep 18 '20 at 15:30

score 0 · Accepted Answer · answered Sep 18 '20 at 14:48

From the documentation (https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM) we see that the inputs need to have a shape of (batch, timesteps, features). Not sure what your dataset looks like, but it sounds like each of the observations in your dataset has a shape of (10,). In this case, you need to combine the observations along a new dimension, time. So the overall dataset should have a shape like (observations, timesteps, features).

rawdata = np.random.rand(1002,10)  # generate random dataset
statemem = 50  # past 50 observations are used to make the next prediction
traindata = []
for i in range(len(rawdata)-statemem):
    traindata.append(rawdata[i:i+statemem,:])
traindata = tf.convert_to_tensor(traindata)

I don't specify the training labels here, but remember each index i in traindata is going to correspond to index i+statemem of the labels. The statemem is the past history you use to make the next lstm prediction. You can treat this as a hyperparameter specific to your problem.

Just one more thing, I have data for 1001 years, and for each year I have observed 10 features in 1002 spatial points, now my batch size should be 1001 or 1002?? — john22, Sep 18 '20 at 15:15
the batch size is independent from the number of observations you have - it is a hyperparameter that you can set to help the optimization algorithm. In general, people use a batch size like 32, 64, or 128. You probably don't want to put every single observation into the batch. The space ('spatial points') vs. time (years) is a more complicated question — Taw, Sep 18 '20 at 15:33

How to reshape to make a 3d input for LSTM?

1 Answers1