Tensorflow CUDA - CUPTI error: CUPTI could not be loaded or symbol could not be found

Question

I use the Tensorflow v 1.14.0. I work on Windows 10. And here is how relevant environment variables look in the PATH:

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\libnvvp
C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common
C:\Users\sinthes\AppData\Local\Programs\Python\Python37
C:\Users\sinthes\AppData\Local\Programs\Python\Python37\Scripts
C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\cuda\bin

Maybe also worth to mention, just in case it might be relevant.. I use Sublime Text 3 for development and I do not use Anaconda. I find it a bit cumbersome to make updates on tensorflow in the conda environment so I just use Sublime Text right now. (I was using Anaconda (Spyder) previously but I uninstalled it from my computer.)

Things seem to work fine except with some occasional strange warnings. But one consistent warning I get is the following whenever I run the fit function.

E tensorflow/core/platform/default/device_tracer.cc:68] CUPTI error: CUPTI could not be loaded or symbol could not be found.

And here is how I call the fit function:

history = model.fit(x=train_x,
                    y=train_y,
                    batch_size=BATCH_SIZE,
                    epochs=110,
                    verbose=2,
                    callbacks=[tensorboard, checkpoint, reduce_lr_on_plateau],
                    validation_data=(dev_x, dev_y),
                    shuffle=True,
                    class_weight=class_weight,
                    steps_per_epoch=None,
                    validation_steps=None)

I just wonder why I see the CUPTI Error message during the run time? It is only printed out once. Is that something that I need to fix or is it something that can be ignored? This message does not tell anything concrete to me to be able to take any action.

score 11 · Answer 1 · edited Aug 21 '19 at 13:27

11

Add this in path for Windows:

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\extras\CUPTI\libx64

edited Aug 21 '19 at 13:27

Laurenz Albe

209,280
17
206
263

answered Aug 21 '19 at 13:05

Pigny Pierre-Olivier

111
2

1

Thanks ! This worked great. Now I have the correct message : "Successfully opened dynamic library cupti64_100.dll" – Ismael EL ATIFI Nov 16 '19 at 13:47
2

I also had to run my Python script as an administrator to make this work. – Adam Haun Nov 25 '19 at 17:50
1

I don't have this path... Any other alternative? – J Agustin Barrachina Jun 19 '20 at 16:24
Following a couple of pratfalls I just experienced: if you're launching from a command window (for instance if you're using Jupyter Lab), remember to start a new command window in order to load the PATH changes; also, you'll need to run the command window as an administrator. I was having problems getting TensorBoard Profiling to work fully using the Keras callback; it's all good now. – Denziloe Dec 03 '20 at 04:58

score 11 · Answer 2 · answered Nov 03 '19 at 18:25

The NVIDIA® CUDA Profiling Tools Interface (CUPTI) is a dynamic library that enables the creation of profiling and tracing tools that target CUDA applications.

CPUTI seems to have been added by the Tensorflow Developors to allow profiling. You can simply ignore the error if you don't mind the exception or adapt your environment path, so the dynamically linked library (DLL) can be found during execution.

Inside of you CUDA installation directory, there is an extras\CUPTI\lib64 directory that contains the cupti64_101.dll that is trying to be loaded. Adding that directory to your path should resolve the issue, e.g.,

SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\extras\CUPTI\lib64;%PATH%

N.B. in case you get an INSUFFICIENT_PRIVILEGES error next, try to run your program as administrator.

score 8 · Answer 3 · answered Nov 07 '19 at 16:17

This answer is for Ubuntu-16.04.

I had this issue when I upgraded to Tensorflow-1.14 with Python2.7 and Python3.6. I had to add /usr/local/cuda/extras/CUPTI/lib64 to LD_LIBRARY_PATH with export LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH and logout and login. source ~/.bashrc didn't help. Note that my cuda folder was pointing to cuda-10.0.

jreeves · Answer 4 · 2021-05-27T23:52:14.950

Ran in to this same issue. This is what fixed it for me incase someone else has a similar problem fixing this.

error I received:

function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.

Windows Server 2019
Tensorflow 2.5
Cuda 11.2 (CUDA_PATH environment variable is set and added to the PATH environment variable)
Cudnn 8.1.0

I had already set C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\extras\CUPTI\lib64 in the PATH environment variable but was still receiving the error.

Running where /r c:\ cupti*.dll in a cmd prompt found dll's in the c:\Program Files\NVIDIA Corporation\Nsight Systems 2020.4.3\target-windows-x64\ directory. Simply adding this directory to the PATH environment variable fixed the error.

score 4 · Answer 5 · answered Jul 03 '19 at 03:44

4

I had a similar error when trying to get tensorboard graph, I think it only affects you if you plan to use tensorboard.

I found the solution in this post but it is for linux https://gist.github.com/Brainiarc7/6d6c3f23ea057775b72c52817759b25c I think you need to create a library configuration file for cupti.

answered Jul 03 '19 at 03:44

MichaelChen

41
2

I do need to be able to use tensorboard. Tensorflow is a mess. – edn Jul 03 '19 at 09:21

score 4 · Answer 6 · edited May 09 '21 at 05:04

Here is what solved "my" problem:

Windows 10, Tensorflow-gpu 2.4

The first issue, was it was unclear about exactly "which" cupti64 version it was trying to load. With that in mind, I did a search for all dll's called cupti*

I then copied them all (yeh I know it's hack, but given limited information...) into my

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\extras\CUPTI\lib64

folder (cupti64_2020.1.0.dll was in there already)

I then needed to also set the folder permission to get it to work, which was strange as I was running VS as admin

score 2 · Accepted Answer · answered Jul 03 '19 at 23:25

Here is what solved "my" problem:

I just replaced my tensorflow v 1.14 with tensorflow v 1.13.1. And no more CUPTI error messages. And even some other strange warnings / problems have disappeared. All issues should obviously have specific reasons but Tensorflow (many times) unfortunately does not provide understandable error/warning messages that give a good/fair idea that helps to solve the issue. And I end up spending hours (even days) on such strange problems, that reduces my productivity significantly.

One general learning for me (that might be relevant to share here) is that I should not be in hurry to upgrade my tensorflow installation to the latest version of it. The latest one is almost never stable, whenever I made a try, I ended up spending significant amount of time on problems that are caused by tensorflow. Poor documentation and error messages make it very very difficult to work with.

If anyone has a better answer, s/he is more than welcome to share his/her insights on the issue I shared in this question.

score 0 · Answer 8 · answered Jul 04 '21 at 17:37

I also just ran into this issue, exactly as jreeves. I solved it exactly following jreeves' method (above). (Thanks to jreeves for his/her work in finding and documenting the solution.) MY setup:

Windows 10
Gpu Support: True
Cuda Support: True
TensorFlow: 2.4.1
Python version: 3.8.8.
Tensorboard version 2.4.1.
Cuda 11.1
Cudnn 8.0.5

Tensorflow CUDA - CUPTI error: CUPTI could not be loaded or symbol could not be found

8 Answers8

Linked