PyTorch calculate MSE and MAE

Question

I would like to calculate the MSE and MAE of the model below. The model is calculating the MSE after each Epoch. What do I need to do to get the overall MSE value, please? Can I use the same code to calculate the MAE? Many Thanks in advance

model.eval()
for images, paths in tqdm(loader_test):
    images = images.to(device)
    targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
    targets = targets.float().to(device)

    # forward pass:
    output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
    preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)

    # logging:
    loss = torch.mean((preds - targets)**2)
    count_error = torch.abs(preds - targets).mean()
    mean_test_error += count_error
    writer.add_scalar('test_loss', loss.item(), global_step=global_step)
    writer.add_scalar('test_count_error', count_error.item(), global_step=global_step)
    
    global_step += 1

average_accuracy = 0 
mean_test_error = mean_test_error / len(loader_test)
writer.add_scalar('mean_test_error', mean_test_error.item(), global_step=global_step)
average_accuracy += mean_test_error
average_accuracy = average_accuracy /len(loader_test)
print("Average accuracy: %f" % average_accuracy)
print("Test count error: %f" % mean_test_error)
if mean_test_error < best_test_error:
    best_test_error = mean_test_error
    torch.save({'state_dict':model.state_dict(),
                'optimizer_state_dict':optimizer.state_dict(),
                'globalStep':global_step,
                'train_paths':dataset_train.files,
                'test_paths':dataset_test.files},checkpoint_path)

Could you clarify what you mean with overall MSE? Is it the MSE that is averaged over all epochs? — cmplx96, Aug 13 '20 at 09:16
Then just sum up the MSE for each epoch and divide by the number of epochs. — cmplx96, Aug 13 '20 at 09:23

Bedir Yilmaz · Accepted Answer · 2020-08-14T04:29:04.590

1

First of all, you would want to keep your batch size as 1 during test phase for simplicity.

This maybe task specific, but calculation of MAE and MSE for a heat map regression model are done based on the following equations:

This means that in your code, you should change the lines where you calculate MAE as following

error = torch.abs(preds - targets).sum().data
squared_error = ((preds - targets)*(preds - targets)).sum().data
runnning_mae += error
runnning_mse += squared_error

and later, after the epoch ends,

mse = math.sqrt(running_mse\len(loader_test))
mae = running_mae\len(loader_test)

edited Aug 14 '20 at 04:29

answered Aug 13 '20 at 09:24

Bedir Yilmaz

3,823
5
34
54

Thanks! In the code above, I have caluclated in a similar way; i think. loss = torch.mean((preds - targets)**2) count_error = torch.abs(preds - targets).mean() Would this not give the same values? For the code after each epoch for the mae =mae = running_mse\len(loader_test) Should this be mae = running_mae\len(loader_test) – Browed1983 Aug 13 '20 at 18:59
For your second question, yes, I have corrected the typo, its should have been mae = running_mae\len(loader_test) – Bedir Yilmaz Aug 14 '20 at 04:31
For your first question, a loss value is computed to be backpropagated through the parameters of the network, therefore you would need its gradients. But an error value is only for evaluation purposes; you do not need its gradient so you can just take its `data` to avoid unnecessary use of gpu memory. See my updated answer. PS: You can also achieve this functionality with `with torch.no_grad():` statement at the top of your test cycle. @Browed1983 – Bedir Yilmaz Aug 14 '20 at 04:36
Thanks for your help. In which format would you say MAE and MSE should be presented in % or as it is? Thanks – Browed1983 Aug 15 '20 at 20:21
1

My pleasure. I would stick with the format used in the influencing papers. This also depends on the dataset. Two digits or in some cases single digit would suffice for the fractional part. Try this: `"MAE: {0:.2f}".format(mae)` – Bedir Yilmaz Aug 15 '20 at 23:09
What do the n and y represent in the equation, please? – Browed1983 Aug 25 '20 at 15:15
1

n is the batch size. y_j is the output (prediction/estimation) and yhat_j is the target (ground truth/label) – Bedir Yilmaz Aug 26 '20 at 13:01

PyTorch calculate MSE and MAE

1 Answers1