4

I've been trying to get an LSTM (LSTM followed by a linear layer in a custom model), working in Pytorch, but was getting the following error when calculating the loss:

Assertion cur_target >= 0 && cur_target < n_classes' failed.

I defined the loss function with:

criterion = nn.CrossEntropyLoss()

and then called with 

loss += criterion(output, target)

I was giving the target with dimensions [sequence_length, number_of_classes], and output has dimensions [sequence_length, 1, number_of_classes].

The examples I was following seemed to be doing the same thing, but it was different on the Pytorch docs on cross entropy loss.

The docs say the target should be of dimension (N), where each value is 0 ≤ targets[i] ≤ C−1 and C is the number of classes. I changed the target to be in that form, but now I'm getting an error saying (The sequence length is 75, and there are 55 classes):

Expected target size (75, 55), got torch.Size([75])

I've tried looking at solutions for both errors, but still can't get this working properly. I'm confused as to the proper dimensions of target, as well as the actual meaning behind the first error (different searches gave very different meanings for the error, none of the fixes worked).

Thanks

LunarLlama
  • 229
  • 2
  • 11

1 Answers1

3

You can use squeeze() on your output tensor, this returns a tensor with all the dimensions of size 1 removed.

This short code uses the shapes you mentioned in your question:

sequence_length   = 75
number_of_classes = 55
# creates random tensor of your output shape
output = torch.rand(sequence_length, 1, number_of_classes)
# creates tensor with random targets
target = torch.randint(55, (75,)).long()

# define loss function and calculate loss
criterion = nn.CrossEntropyLoss()
loss = criterion(output, target)
print(loss)

Results in the error you described:

ValueError: Expected target size (75, 55), got torch.Size([75])

So using squeeze() on your output tensor solves your problem by getting it to correct shape.

Example with corrected shape:

sequence_length   = 75
number_of_classes = 55
# creates random tensor of your output shape
output = torch.rand(sequence_length, 1, number_of_classes)
# creates tensor with random targets
target = torch.randint(55, (75,)).long()

# define loss function and calculate loss
criterion = nn.CrossEntropyLoss()

# apply squeeze() on output tensor to change shape form [75, 1, 55] to [75, 55]
loss = criterion(output.squeeze(), target)
print(loss)

Output:

tensor(4.0442)

Using squeeze() changes your tensor shape from [75, 1, 55] to [75, 55] so it that output and target shape matches!

You can also use other methods to reshape your tensor, it is just important that you have the shape of [sequence_length, number_of_classes] instead of [sequence_length, 1, number_of_classes].

Your targets should be a LongTensor resp. a tensor of type torch.long containing the classes. Shape here is [sequence_length].

Edit:
 Shapes from above example when passing to cross-entropy function:

Outputs: torch.Size([75, 55])
Targets: torch.Size([75])

Here is a more general example what outputs and targets should look like for CE. In this case we assume we have 5 different target classes, there are three examples for sequences of length 1, 2 and 3:

# init CE Loss function
criterion = nn.CrossEntropyLoss()

# sequence of length 1
output = torch.rand(1, 5)
# in this case the 1th class is our target, index of 1th class is 0
target = torch.LongTensor([0])
loss = criterion(output, target)
print('Sequence of length 1:')
print('Output:', output, 'shape:', output.shape)
print('Target:', target, 'shape:', target.shape)
print('Loss:', loss)

# sequence of length 2
output = torch.rand(2, 5)
# targets are here 1th class for the first element and 2th class for the second element
target = torch.LongTensor([0, 1])
loss = criterion(output, target)
print('\nSequence of length 2:')
print('Output:', output, 'shape:', output.shape)
print('Target:', target, 'shape:', target.shape)
print('Loss:', loss)

# sequence of length 3
output = torch.rand(3, 5)
# targets here 1th class, 2th class and 2th class again for the last element of the sequence
target = torch.LongTensor([0, 1, 1])
loss = criterion(output, target)
print('\nSequence of length 3:')
print('Output:', output, 'shape:', output.shape)
print('Target:', target, 'shape:', target.shape)
print('Loss:', loss)

Output:

Sequence of length 1:
Output: tensor([[ 0.1956,  0.0395,  0.6564,  0.4000,  0.2875]]) shape: torch.Size([1, 5])
Target: tensor([ 0]) shape: torch.Size([1])
Loss: tensor(1.7516)

Sequence of length 2:
Output: tensor([[ 0.9905,  0.2267,  0.7583,  0.4865,  0.3220],
        [ 0.8073,  0.1803,  0.5290,  0.3179,  0.2746]]) shape: torch.Size([2, 5])
Target: tensor([ 0,  1]) shape: torch.Size([2])
Loss: tensor(1.5469)

Sequence of length 3:
Output: tensor([[ 0.8497,  0.2728,  0.3329,  0.2278,  0.1459],
        [ 0.4899,  0.2487,  0.4730,  0.9970,  0.1350],
        [ 0.0869,  0.9306,  0.1526,  0.2206,  0.6328]]) shape: torch.Size([3, 5])
Target: tensor([ 0,  1,  1]) shape: torch.Size([3])
Loss: tensor(1.3918)

I hope this helps!

MBT
  • 21,733
  • 19
  • 84
  • 102
  • I changed the dimensions as you said, but am now getting a different error. I think I am misunderstanding the form the target should take. It says multi-target not supported at c:\programdata\miniconda3\conda-bld\pytorch_1533086652614\work\aten\src\thnn\generic/ClassNLLCriterion.c:21 My target tensor is given as a one-hot encoding, showing what each character is. Is this how it should be set up? I saw answers saying it should be given as class indices, but is that not what I initially tried going off the Pytorch docs that gave a dimension error? Thanks for the detailed answer. – LunarLlama Nov 24 '18 at 17:15
  • 1
    @LunarLlama Yes, it can be confusing. You can try out the 2nd example in my answer, check the shapes and apply it to your own program. Your target should be a LongTensor and have shape `[75]`. – MBT Nov 24 '18 at 17:24
  • I think I've gotten the dimensions sorted out with your help, but am still confused on what goes into the target tensor. Is it the index of the correct class/ index of the correct character? Thanks for all the help. – LunarLlama Nov 24 '18 at 17:31
  • 1
    In your outputs you have each element of the sequence a distribution over the given classes, thus `[sequence_length, number_of_classes]` in your targets you just have an index value for the respective class for each element hence it is of shape `[sequence_length]`. Does this make sense to you? – MBT Nov 24 '18 at 17:45
  • 1
    @LunarLlama I edited my answer and added a more general example at the end, I hope this helps understanding the shapes! If so it would be great if you could accept the answer :) – MBT Nov 24 '18 at 18:09
  • 1
    The network runs! Thank you so much, I really appreciate all your detailed answers. – LunarLlama Nov 24 '18 at 18:40
  • Sorry, one more question. I've been using this for batch size of 1, what size would the target tensor be for different batch sizes? I tried having the target tensor be of size [seq_length, batch_size] but get an error. The input and output tensors are both now of sizes [seq_length, batch_size, num_classes], as per the docs. – LunarLlama Nov 26 '18 at 02:22
  • 1
    @LunarLlama I would just flatten out your tensors so: `outputs.view(seq_length* batch_size, num_classes)`, `targets.view(-1)`. I think this is also the intended way as on the docs it says: `Input: (N,C)` *where C = number of classes* and `Target: (N)`. So `view` commands above should give you that shape. – MBT Nov 26 '18 at 08:34