0

I use theano 0.9.0, CUDA 8.0.61, gcc version 4.9.3, NVIDIA UNIX x86_64 Kernel Module 375.39, Ubuntu 16.04.2 LTS and I have a geforce 1060. CUDA works as I have done the tests.

But whenever I do the gpu test with theano (http://deeplearning.net/software/theano/tutorial/using_gpu.html) I get this error : ImportError: ('The following error happened while compiling the node', GpuElemwise{exp,no_inplace}((float64, vector)>), '\n', 'libnvrtc.so.7.5: cannot open shared object file: No such file or directory', '[GpuElemwise{exp,no_inplace}((float64, vector)>)]')

Which is really silly as I have CUDA 8, not CUDA 7.5 so of course it cannot find libnvrtc.so.7.5. How can I tell theano that I'm using CUDA 8 and not CUDA 7.5 ?

I have this overly complicated setup to my .bashrc :

  • export PATH="/usr/local/cuda-8.0/bin:$PATH"
  • export LD_LIBRARY_PATH="/usr/local/cuda-8.0/lib64"
  • export CUDA_HOME=/usr/local/cuda-8.0
  • export CUDA_ROOT=/usr/local/cuda-8.0
  • export CPATH="$CPATH:~/.local/include"
  • export CPATH="/usr/local/cuda-8.0/include:$CPATH"
  • export LIBRARY_PATH="$LIBRARY_PATH:~/.local/lib"
  • export LIBRARY_PATH="/usr/lib/nvidia-375:$LIBRARY_PATH"
  • export LIBRARY_PATH="/usr/local/cuda-8.0/lib64:$LIBRARY_PATH"
  • export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:~/.local/lib"

and my .theanorc is :

[nvcc]

flags=-D_FORCE_INLINES

[global]

device = cuda0

[cuda]

root = /usr/local/cuda-8.0/

Alexia
  • 3
  • 1

2 Answers2

1

You can't. Either install CUDA 7.5 or install a version of the framework built against CUDA 8. There is no way to make what you are trying to do work.

talonmies
  • 70,661
  • 34
  • 192
  • 269
  • Thanks for the help. What do you mean by "framework built against CUDA 8"? I guess I'll uninstall CUDA 8 and install 7.5, I really thought CUDA 8 worked with theano as many guides make you download CUDA 8. – Alexia Apr 09 '17 at 16:20
  • Framework = Theano in this case. It isn't a question of whether Theano is compatible with CUDA 8, because it is. It is a question of whether the version you have was built against CUDA 8, and it clearly wasn't – talonmies Apr 09 '17 at 17:16
  • Just checked and CUDA 7.5 is not available for Ubuntu 16.04 (https://developer.nvidia.com/cuda-75-downloads-archive). I'm not sure what to do. I tried reinstalling sckit-cuda and did some tests which it passed but theano still give the same error. To answer your last comment, I downloaded the bleeding-edge version, so it should be compatible but let me recheck. Edit: Okay I must build theano against CUDA 8.0 but what does it means? How does one install theano for CUDA 8.0 is what I don't understand. – Alexia Apr 09 '17 at 17:17
  • I did sudo pip install git+https://github.com/Theano/Theano.git#egg=Theano as recommended here : http://deeplearning.net/software/theano/install_ubuntu.html, how can I specify CUDA 8 in the install? – Alexia Apr 09 '17 at 17:24
  • As i have now said three times, you can't. This is a binary compatibility question. You must have the same version of the CUDA runtime library and tools as your Theano instance was compiled against. – talonmies Apr 09 '17 at 17:39
-1

Edit: You should perhaps not insist on trying to make this work, given that your card is Pascal-generation and CUDA 7.5 is pre-Pascal. With that in mind...

While @talonmies' answer is correct, at least one (and probably more) of the following things should actually get Theano working for you:

  1. Add a package repository for newer versions of CUDA targetted at *buntu 16.04 Xenial:

    deb http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu xenial main
deb-src http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu xenial main

    put these file named, say /etc/apt/sources.list.d/graphics-drivers-ppa-xenial.list - but remember to remove/update it if you upgrade to newer version of *buntu. This should give you access to CUDA 7.5. Make sure not to install an older nVIDIA driver version, though.

  2. Manually install CUDA 7.5. You can download it from here. Adjust the settings mentioned in your question to point at /usr/local/cuda-7.5. Again, make sure you don't also install an older nVIDIA driver.

  3. Follow the installation instructions here. I'm not sure there's much more in there other than using pip, but maybe there is.

Caveat: I've never used Theano myself.

einpoklum
  • 118,144
  • 57
  • 340
  • 684
  • 1
    Using CUDA 7.5 with Pascal GPUs isn't a good idea. And if your method happens to try to pull in a CUDA 7.5 driver, that's a recipe for disaster. – Robert Crovella Apr 09 '17 at 23:22
  • @RobertCrovella: Edited accordingly. However, other than the driver issue, I'm not sure why you state that using CUDA 7.5 would be a bad idea, other than not being able to use Pascal-specific features. – einpoklum Apr 09 '17 at 23:44
  • 1
    0. You'll still need to have a CUDA 8 capable driver installed, which can be challenging for some users 1. any scripts that are built to target pascal (`nvcc -arch=sm_6x ...`) will break. 2. Any usage of Pascal will be via PTX JIT only. Large libraries can have substantial JIT delays, etc. 3. any calls to curand will [just fail](https://devtalk.nvidia.com/default/topic/941354/all-curand-samples-fail-with-gtx-1080-on-cuda-7-5/). 4. NVIDIA [recommends](https://developer.nvidia.com/cuda-toolkit) CUDA 8 with Pascal 5. there must be 50 ways to leave your lover. – Robert Crovella Apr 10 '17 at 00:17
  • @RobertCrovella: Point no 3. is convincing, the rest seems to be obvious and not really a problem in OP's case. – einpoklum Apr 10 '17 at 06:48
  • I tried CUDA 7.5 but it didn't work and then I even reinstalled with Ubuntu 14.04 hoping this would help since CUDA 7.5 is made to work for Ubuntu 14 but even more problems arised. I'm wondering, if the latest theano version is so outdated still using CUDA 7.5, would you recommend other libraries that work well with CUDA 8 and Pascal gpus? Maybe tensorflow or torch? I'm fed up with trying to get theano working. – Alexia Apr 11 '17 at 12:05
  • @Alexia: Unfortunately, I can't recommend anything, since I'm not a Python person. Consider asking the same question on http://softwarerecs.stackexchange.com or on http://devtalk.nvidia.com . – einpoklum Apr 11 '17 at 12:35