Pytorch: multi-target error with CrossEntropyLoss

Question

So I was training a Conv. Neural Network. Following are the essential details:

original label dim = torch.Size([64, 1])
output from the net dim = torch.Size([64, 2])
loss type = nn.CrossEntropyLoss()
error = RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15

WHERE AM I WRONG..?

training:

EPOCHS        = 5
LEARNING_RATE = 0.0001
BATCH_SIZE    = 64

net = Net().to(device)
optimizer = optim.Adam(net.parameters(), lr=LEARNING_RATE)

loss_log = []
loss_log = train(net, trainSet, loss_log, EPOCHS, LEARNING_RATE, BATCH_SIZE)

train function:

def train(net, train_set, loss_log=[], EPOCHS=5, LEARNING_RATE=0.001, BATCH_SIZE=32):
  print('Initiating Training..')  
  loss_func = nn.CrossEntropyLoss()

  # Iteration Begins
  for epoch in tqdm(range(EPOCHS)):
    # Iterate over every sample in the batch
    for data in tqdm(trainSet, desc=f'Iteration > {epoch+1}/{EPOCHS} : ', leave=False):
        x, y = data
        net.zero_grad()

        #Compute the output
        output, sm = net(x)

        # Compute Train Loss
        loss = loss_func(output, y.to(device))

        # Backpropagate
        loss.backward()

        # Update Parameters
        optimizer.step()

        # LEARNING_RATE -= LEARNING_RATE*0.0005

    loss_log.append(loss)
    lr_log.append(LEARNING_RATE)

  return loss_log, lr_log

FULL ERROR:

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-20-8deb9a27d3b4> in <module>()
     13 
     14 total_epochs += EPOCHS
---> 15 loss_log = train(net, trainSet, loss_log, EPOCHS, LEARNING_RATE, BATCH_SIZE)
     16 
     17 plt.plot(loss_log)

4 frames

<ipython-input-9-59e1d2cf0c84> in train(net, train_set, loss_log, EPOCHS, LEARNING_RATE, BATCH_SIZE)
     21         # Compute Train Loss
     22         # print(output, y.to(device))
---> 23         loss = loss_func(output, y.to(device))
     24 
     25         # Backpropagate

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
    914     def forward(self, input, target):
    915         return F.cross_entropy(input, target, weight=self.weight,
--> 916                                ignore_index=self.ignore_index, reduction=self.reduction)
    917 
    918 

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2019     if size_average is not None or reduce is not None:
   2020         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2021     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2022 
   2023 

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1836                          .format(input.size(0), target.size(0)))
   1837     if dim == 2:
-> 1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1839     elif dim == 4:
   1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15

Likely an issue with how you've defined `y`. If `x` is a size `[B, C]` tensor then `y` should be a LongTensor of size `[B]`. — jodag, Apr 01 '20 at 00:08
y is already a longTensor. This is how I've defined y while creating a positive label : y = torch.Tensor([1]).long() {NOTE: I created the Batch after this, thus the before mentioned shape} — Parthik B., Apr 01 '20 at 00:33
This particular error occurs if the sizes of x and y are inconsistent, so take a look at `x.size()` and `y.size()` right before you call the loss function. — jodag, Apr 01 '20 at 06:44

score 6 · Answer 1 · answered Apr 01 '20 at 16:31

6

The problem is that your target tensor is 2-dimensional ([64,1] instead of [64]), which makes PyTorch think that you have more than 1 ground truth label per data. This is easily fixed via loss_func(output, y.flatten().to(device)). Hope this helps!

answered Apr 01 '20 at 16:31

ccl

2,378
2
12
26

i believe y.squeeze(1) will also do the trick – Jimmy TwoCents May 12 '21 at 19:04

Edoardo Guerriero · Answer 2 · 2020-04-01T01:06:15.153

-1

You wrote yourself the problem:

original label dim = torch.Size([64, 1]) <-- [0] or [1]
output from the net dim = torch.Size([64, 2]) <-- [0,1] or [1,0]

You need to change your target into one hot encoding. Moreover, if you're doing a binary classification I would suggest to change the model to return a single output unit and use binary_cross_entropy as a loss function.

edited Apr 01 '20 at 01:06

answered Apr 01 '20 at 00:54

Edoardo Guerriero

1,210
7
16

I don't think I need to do all that stuff because I'm using CrossEntropyLoss and it does all that stuff itself. I'll check and verify once again anyways. – Parthik B. Apr 01 '20 at 02:50

Pytorch: multi-target error with CrossEntropyLoss

2 Answers2