ROC curve and AUC for a multiclass and multilabel problem. ValueError: Target scores need to be probabilities for multiclass roc_auc

Question

so, I would like to calculate the ROC curve and AUC of a code of mine where I have 28 classes and my images can be several at the same time. For example, an image can belong to class 1, 2 and 3 at the same time. I have a vector of 28 positions as a label in y_true and there it is marked with 1 in the position for the class. For example, if an image belongs to class 2, 3 and 5 in positions 2, 3 and 5 of the vector, they will be marked with 1 -> [0,0,1,1,0,1,0,0,0 ,..., 0]

def data_validate(samples, loss, network, f1_class):

x, y_true = samples #x-->valor na matriz, y --> label
x = x.cuda() #to GPU
y_true = y_true.cuda() #to GPU
y_pred = network(x) #aqui executa o forward do model.py dos {batch_size} e retorna o fc
y_pred = torch.sigmoid(y_pred)
erro = loss(y_pred, y_true)
f1_class.acumulate(y_pred.cpu().detach(), y_true.cpu().detach(),th=0.5)
print(y_pred)
for i in range(28):
    auc_score = roc_auc_score(y_true[:][i].cpu().detach(), y_pred.cpu().detach(), multi_class='ovr')

return erro, y_pred.cpu().detach(), y_true.cpu().detach()

but I receive that error --> ValueError: Target scores need to be probabilities for multiclass roc_auc, i.e. they should sum up to 1.0 over classes

Answer 1

You can see in PyTorch documentation that torch.sigmoid() output is not guaranteed to sum to 1 over classes.

>>> a = torch.randn(4)
>>> a
tensor([ 0.9213,  1.0887, -0.8858, -1.7683])
>>> torch.sigmoid(a)
tensor([ 0.7153,  0.7481,  0.2920,  0.1458])

You need to normalize y_pred to solve this error.

Code for doing it could be found here. You need something like:

row_sums = torch.sum(y_pred, 1) # normalization 
row_sums = row_sums.repeat(1, num_classes) # expand to same size as out
y_pred = torch.div( y_pred , row_sums ) # these should be histograms

Answer 2

Code changed to:

def data_validate(samples, loss, network, f1_class):
    x, y_true = samples #x-->valor na matriz, y --> label
    x = x.cuda() #to GPU
    y_true = y_true.cuda() #to GPU
    y_pred = network(x) #aqui executa o forward do model.py dos {batch_size} e retorna o fc
    y_pred = torch.sigmoid(y_pred)
    erro = loss(y_pred, y_true)
    f1_class.acumulate(y_pred.cpu().detach(), y_true.cpu().detach(),th=0.5)

    row_sums = torch.sum(y_pred, 1)
    row_sums = row_sums.repeat(28)
    row_sums = row_sums.reshape(y_pred.shape)
    y_pred = torch.div( y_pred , row_sums ) 

    for i in range(len(y_pred)):
        auc_score = roc_auc_score(y_true[:][i].cpu().detach(), y_pred[:][i].cpu().detach(), multi_class='ovr')

    return erro, y_pred.cpu().detach(), y_true.cpu().detach()

Like Priya said.