Multi-target loss recommendations

Question

I'm working on a classification problem. The number of classes is 5. I have a ground truth vector that has the shape (3) instead of 1. The values in this target vector are the possible classes and the predicted vector is of the shape (1x5) which holds the softmax scores for all the classes.

For example:

predicted_vector = tensor([0.0669, 0.1336, 0.3400, 0.3392, 0.1203]
ground_truth = tensor([3,2,5])

For the above illustration, a typical argmax operation would result in declaring class 3 as the predicted class (0.34) but I want the model to reward even if the argmax class is any of 3,2, or 5.

Which loss function is recommended for such a use case?

This seems like it would work better if posed as a standard multi-label classification problem. That would usually mean using `nn.BCEWithLogitsLoss` instead of `nn.BinaryCrossEntropy`. In that case you would use [0,1,1,0,1] as your target vector (one-hot encoding of your labels) and you could interpret the sigmoid of the model logits as the probability of the associated label. — jodag, Oct 04 '22 at 14:30

score 1 · Answer 1 · answered Oct 04 '22 at 23:25

As jodag pointed out in the comments you can try to treat it as a multi-label classification problem.

So [[0, 1, 2], [0, 2, 4], [3, 3, 3]] will be transformed into:

tensor([[1., 1., 1., 0., 0.],
        [1., 0., 1., 0., 1.],
        [0., 0., 0., 1., 0.]])

Here is an example of how this can be implemented:

import torch
from torch.nn import BCELoss

predicted_vector = torch.rand((3, 5))
ground_truth = torch.LongTensor([[0, 1, 2], [0, 2, 4], [3, 3, 3]])

labels_onehot = torch.zeros_like(predicted_vector)
labels_onehot.scatter_(1, ground_truth, 1)

loss_fn = BCELoss()
loss = loss_fn(predicted_vector, labels_onehot)

Also you can add different weights to different labels

Hi @u1234x1234, I've uploaded the solution for my specific use case — helloworld, Oct 05 '22 at 06:18

score 0 · Accepted Answer · answered Oct 05 '22 at 06:17

For this problem, a given sample is in exactly one class (say, class 3), but for training purposes, predicting class 2 or 5 is still okay so the model isn't penalised that heavily.

This is a typical single-label, multi-class problem, but with probabilistic (“soft”) labels, and CrossEntropyLoss should be used (and not use softmax()).

In this example, the (soft) target might be a probability of 0.7 for class 3, a probability of 0.2 for class 2, and a probability of 0.1 for class 5 (and zero for everything else).

Multi-target loss recommendations

2 Answers2