Speeding up the trainning - RNN with LSTM in PyTorch

Question

I am trying to train a LSTM for energy demand forecast but it takes too long. I do not understand why because the model looks “simple” and there is no much data. Might it be because I am not using the DataLoader? How could I use it with RNN since I have a sequence?

Complete code is in Colab: https://colab.research.google.com/drive/130rG8_j1Lf8RQoVRrfXCeo5h_CcC5NU6?usp=sharing

The interesting part to be improved may be this:

for seq, y_train in train_data:
    optimizer.zero_grad()
    model.hidden = (torch.zeros(1,1,model.hidden_size),
                    torch.zeros(1,1,model.hidden_size))
    y_pred = model(seq)
    loss = criterion(y_pred, y_train)
    loss.backward()
    optimizer.step()

Thanks in advance to anyone helping me.

Thanks @GirishDattatrayHegde, that is true but the dataset is small enough and other models run faster being more complex. — jccarrasco, Aug 25 '20 at 11:03
What do you mean by "more complex" models? I mean, which models are faster than RNNs in your end? Cause, RNNs if are not the most complicated neural models, but definitely are among such "complex" models due to their recurrent nature. @jccarrasco — inverted_index, Aug 26 '20 at 03:27
What do you mean by too long? What is the time for one epoch? — Dimitri Sifoua, Aug 26 '20 at 05:09
I think you should use other optimizer like Adam or RMSprop which are much more rapid than SGD optimizer. — Dimitri Sifoua, Aug 26 '20 at 05:11
Thanks @inverted_index, the problem was located in the batch loading. I was passing just 1 batch per training.I will write the solution. — jccarrasco, Aug 26 '20 at 07:54
Thanks @DimitriK.Sifoua. The problem was in the dataloader. I will answer the question. — jccarrasco, Aug 26 '20 at 07:55
Please@DimitriK.Sifoua vote the solution up if you found it useful. — jccarrasco, Aug 26 '20 at 08:12
Please@GirishDattatrayHegde vote the solution up if you found it useful. — jccarrasco, Aug 26 '20 at 08:12
Please@inverted_index vote the solution up if you found it useful. — jccarrasco, Aug 26 '20 at 08:12

jccarrasco · Accepted Answer · 2020-08-26T09:52:22.910

Should you want to speed up the process of training, more data must be provided to the model per training. In my case I was providing just 1 batch. The best way to simply solve this is using the DataLoader.

Complete Colab with the solution can be found in this link: https://colab.research.google.com/drive/1QgtshCFETZ9oTvIYWy1Bdre-614kbwRX?usp=sharing

# This is to create the Dataset
from torch.utils.data import Dataset, DataLoader

class DemandDataset(Dataset):
    def __init__(self, X_train, y_train):
        self.X_train = X_train
        self.y_train = y_train

    def __len__(self):
        return len(self.y_train)

    def __getitem__(self, idx):
        data = self.X_train[idx]
        labels = self.y_train[idx]
        return data, labels

#This is to convert from typical RNN sequences
sq_0 =[]
y_0 =[]
for seq, y_train in train_data:
  sq_0.append(seq)
  y_0.append(y_train)

dataset=DemandDataset(sq_0,y_0)
dataloader = DataLoader(dataset, batch_size=20)

epochs = 30
t = 50

for i in range(epochs):
    print("New epoch")    
  
    for data,label in dataloader:
         
        optimizer.zero_grad()
        model.hidden = (torch.zeros(1,1,model.hidden_size),
                        torch.zeros(1,1,model.hidden_size))
        
        y_pred = model(seq)
        
        loss = criterion(y_pred, label)
        loss.backward()
        optimizer.step()
        
   
    print(f'Epoch: {i+1:2} Loss: {loss.item():10.8f}')
    
   
    preds = train_set[-window_size:].tolist()

    for f in range(t):  
        seq = torch.FloatTensor(preds[-window_size:])
        with torch.no_grad():
            model.hidden = (torch.zeros(1,1,model.hidden_size),
                            torch.zeros(1,1,model.hidden_size))
            preds.append(model(seq).item())
            
    loss = criterion(torch.tensor(preds[-window_size:]),y[-t:])

Speeding up the trainning - RNN with LSTM in PyTorch

1 Answers1