Pytorch fails with CUDA error: device-side assert triggered on Colab

Question

I am trying to initialize a tensor on Google Colab with GPU enabled.

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

t = torch.tensor([1,2], device=device)

But I am getting this strange error.

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1

Even by setting that environment variable to 1 seems not showing any further details.
Anyone ever had this issue?

Answer 1

While I tried your code, and it did not give me an error, I can say that usually the best practice to debug CUDA Runtime Errors: device-side assert like yours is to turn collab to CPU and recreate the error. It will give you a more useful traceback error.

Most of the time CUDA Runtime Errors can be the cause of some index mismatching so like you tried to train a network with 10 output nodes on a dataset with 15 labels. And the thing with this CUDA error is once you get this error once, you will recieve it for every operation you do with torch.tensors. This forces you to restart your notebook.

I suggest you restart your notebook, get a more accuracate traceback by moving to CPU, and check the rest of your code especially if you train a model on set of targets somewhere.

Answer 2

As the other respondents indicated: Running it on CPU reveals the error. My target labels where {1,2} I changed them to {0,1}. This procedure solved it for me.

Answer 3

Double-check the number of gpu. Normally, it should be gpu=0 unless you have more than one gpu.

Answer 4

Bumped into the same issue when using Transformers Trainer. In my case, the issue was caused by model input and tokenizer length sizes mismatch. Here's what solved the issue for me:

model.resize_token_embeddings(len(tokenizer))

and mismatch was caused when adding pad token:

tokenizer.add_special_tokens({'pad_token': '<pad>'})

Answer 5

1st time:

Got the same error while using simpletransformers library to fine-tuning transformer-based model for multi-class classification problem. simpletransformers is a library written on the top of transformers library.

I changed my labels from string representations to numbers and it worked.

2nd time:

Face the same error again while training another transformer-based model with transformers library, for text classification. I had 4 labels in the dataset, named 0,1,2, and 3. But in the last layer (Linear Layer) of my model class, I had two neurons. nn.Linear(*, 2)* which I had to replace by nn.Linear(*, 4) because I had total four labels.

Answer 6

I had the same problem on Colab as well. If your code runs normally on device("cpu"), try deleting the current Colab runtime and restart it. This worked for me.

Answer 7

Maybe, I mean in some cases

It is due to you forgetting to add a sigmoid activation before you send the logit to BCE Loss.

Hope it can help :P

Answer 8

I also encountered this problem and found the reason, because the vocabulary dimension is 8000, but the embedding dimension in my model is set to 5000

Answer 9

This is an open-ended question for most people who land on this page because the underlying issue is different in each case. In my case the error appeared on Colab when I tried to run this notebook on Colab pro: https://colab.research.google.com/drive/1SRclU2pcgzCkVXpmhKppVbGW4UcCs5xT?usp=sharing at supervised_finetuning_trainer.train() step.

If there's someone like me who could not bring the computation into CPU instead of GPU (mostly because the error stack-trace led to a different package like transformers, ..., leading all the way back to pytorch), here's the approach to get a more accurate stack-trace:

https://github.com/huggingface/transformers/blob/ad78d9597b224443e9fe65a94acc8c0bc48cd039/docs/source/en/troubleshooting.md?plain=1#L110

Credits: sgugger on GitHub.

Answer 10

I am a filthy casual coming from the VQGAN+Clip "ai-art" community. I get this error when I already have a session running on another tab. Killing all sessions from the session manager clears it up, and let's you connect with the new tab, which is nice if you have fiddled with a lot of settings you don't want to loose

Answer 11

In my case, I first tried to run my computations on the CPU to detect the actual issue. It turned out that my image transforms were wrong I was applying some un-necessary transformation to my mask image

Answer 12

I also ran into a similar error, and the problem was with the label mismatch only! My train set and test set had different label counts and thus, this error was coming.