Pytorch: Weight in cross entropy loss

Question

I was trying to understand how weight is in CrossEntropyLoss works by a practical example. So I first run as standard PyTorch code and then manually both. But the losses are not the same.

from torch import nn
import torch
softmax=nn.Softmax()
sc=torch.tensor([0.4,0.36])
loss = nn.CrossEntropyLoss(weight=sc)
input = torch.tensor([[3.0,4.0],[6.0,9.0]])
target = torch.tensor([1,0])
output = loss(input, target)
print(output)
>>1.7529

Now for manual Calculation, first softmax the input:

print(softmax(input))
>>
tensor([[0.2689, 0.7311],
        [0.0474, 0.9526]])

and then negetive log of the correct class probality and multiply with the respective weight:

((-math.log(0.7311)*0.36) - (math.log(0.0474)*0.4))/2
>>
0.6662

What I am missing here?

Prajot Kuvalekar · Answer 1 · 2021-07-07T13:48:00.407

14

To compute class weight of your classes use sklearn.utils.class_weight.compute_class_weight(class_weight, *, classes, y) read it here
This will return you an array i.e weight.
eg .

x = torch.randn(20, 5) 
y = torch.randint(0, 5, (20,)) # classes
class_weights=class_weight.compute_class_weight('balanced',np.unique(y),y.numpy())
class_weights=torch.tensor(class_weights,dtype=torch.float)
 
print(class_weights) #([1.0000, 1.0000, 4.0000, 1.0000, 0.5714])

Then pass it to nn.CrossEntropyLoss's weight variable

criterion = nn.CrossEntropyLoss(weight=class_weights,reduction='mean')

loss = criterion(...)

edited Jul 07 '21 at 13:48

answered Jun 04 '21 at 11:50

Prajot Kuvalekar

5,128
3
21
32

should `np.unique(y)` be ordered ascendingly? – Minions Aug 05 '22 at 20:49
Those will be your number of classes; yes by default it will be ascending order – Prajot Kuvalekar Aug 07 '22 at 05:01
2

Shouldn't you pass `1/class_weights`? After all you want to increase the weight for minority classes. – Alaa M. Sep 04 '22 at 06:57
1

Code returns error "compute_class_weight() takes 1 positional argument but 3 were given" – Isaac Zhao Nov 14 '22 at 19:05
@IsaacZhao you need to pass arguments implicitly `compute_class_weight('balanced', classes=np.unique(y), y=y.numpy())` – pashok3ddd Jan 12 '23 at 15:41
1

@AlaaM. `sklearn.utils.class_weight.compute_class_weight` actually calculates inverse frequency as you expected: `len(y) / np.bincount(y)` -- so minority class gets a higher weight – tozCSS Jun 02 '23 at 22:02
@tozCSS - You are right! (but to be accurate, they further normalize by `n_classes` (number of classes)) – Alaa M. Jun 04 '23 at 11:26

score 4 · Answer 2 · answered Apr 24 '20 at 19:13

4

For any weighted loss (reduction='mean'), the loss will be normalized by the sum of the weights. So in this case:

((-math.log(0.7311)*0.36) - (math.log(0.0474)*0.4))/(.4+.36)
>> 1.7531671457872036

answered Apr 24 '20 at 19:13

user3363813

567
1
5
19

Pytorch: Weight in cross entropy loss

2 Answers2

Linked