0

I want to use mlp do prediction(98 vectors of 100), when I save the results, there only 7 vectors(batch=10, the output of one batch). Even if I set batch=90, I saved 7 vectors. I want to know how could I save all baches results?

I save it as followed:

length=len(X) // batch_size
print(length)
for epoch in range(total_epoch):
    loss_sum=0.0
    for batch_idx in range(length+1):
        if batch_idx != length:
            x_ = X[batch_idx*batch_size : (batch_idx+1)*batch_size]
            y_ = Y[batch_idx*batch_size : (batch_idx+1)*batch_size]
        else:
            x_ = X[batch_idx*batch_size : ]
            y_ = Y[batch_idx*batch_size : ]

        x_proj = projector(x_)
        loss = torch.nn.MSELoss()(x_proj, y_)
        loss_sum += loss.item()
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        if batch_idx % print_freq == 0:
            print(f"Epoch: {epoch}, Iteration: {batch_idx}, Loss: {loss.item():.4f}")



    print(f"Epoch: {epoch}, Loss Avg: {loss_sum/length: .4f}")
    print('pred:',x_proj)


pred=x_proj.detach().cpu().numpy()  # Append batch output to list
out_pred = pd.DataFrame(pred).to_csv('prediction+{batch_idx}.csv')  

out_pred = pd.DataFrame(pred).to_csv('prediction.csv')
kkk123
  • 11
  • 2

1 Answers1

0

What seems to be the issue is that you are only saving the last batch of the data loader, and it would appear like it is incomplete (7 out of 10 elements in the batch).

You need to indent your statement in order to save the batches for every batch_idx. Inside your for batch_idx in range(length+1): loop, you can have a condition to check whether the epoch is the last one:

        if epoch == total_epoch - 1:
            pred=x_proj.detach().cpu().numpy()  # Append batch output to list
            out_pred = pd.DataFrame(pred).to_csv('prediction+{batch_idx}.csv')
Ivan
  • 34,531
  • 8
  • 55
  • 100