cudnn compile configuration in TensorFlow

Question

Ubuntu 14.04, CUDA Version 7.5.18, nightly build of tensorflow

While running a tf.nn.max_pool() operation in tensorflow, I got the following error:

E tensorflow/stream_executor/cuda/cuda_dnn.cc:286] Loaded cudnn library: 5005 but source was compiled against 4007. If using a binary install, upgrade your cudnn library to match. If building from sources, make sure the library loaded matches the version you specified during compile configuration.

W tensorflow/stream_executor/stream.cc:577] attempting to perform DNN operation using StreamExecutor without DNN support

Traceback (most recent call last):

...

How do I specify my cudnn version in the compile configuration of tensorflow?

I'm confused what this error even means, does it mean that you have a version of cudnn but tensorflow expected a different one or what does it mean? — Charlie Parker, Oct 10 '16 at 18:02

score 2 · Accepted Answer · answered Jun 06 '16 at 17:08

Go in the directory of TensorFlow source code, then execute the configuration file: /.configure.

Here is an example from the TensorFlow documentation:

$ ./configure
Please specify the location of python. [Default is /usr/bin/python]:
Do you wish to build TensorFlow with GPU support? [y/N] y
GPU support will be enabled for TensorFlow

Please specify which gcc nvcc should use as the host compiler. [Default is
/usr/bin/gcc]: /usr/bin/gcc-4.9

Please specify the Cuda SDK version you want to use, e.g. 7.0. [Leave
empty to use system default]: 7.5

Please specify the location where CUDA 7.5 toolkit is installed. Refer to
README.md for more details. [default is: /usr/local/cuda]: /usr/local/cuda

Please specify the Cudnn version you want to use. [Leave empty to use system
default]: 4.0.4

Please specify the location where the cuDNN 4.0.4 library is installed. Refer to
README.md for more details. [default is: /usr/local/cuda]: /usr/local/cudnn-r4-rc/

Please specify a list of comma-separated Cuda compute capabilities you want to
build with. You can find the compute capability of your device at:
https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your
build time and binary size. [Default is: \"3.5,5.2\"]: 3.5

Setting up Cuda include
Setting up Cuda lib64
Setting up Cuda bin
Setting up Cuda nvvm
Setting up CUPTI include
Setting up CUPTI lib64
Configuration finished

So there is no way around having to recompile tensorflow yourself with bazel, in order to change the cudnn version? I cannot configure my current installation? — Alexander R Johansen, Jun 06 '16 at 17:32
When running `./configure`, you will have to recompile with bazel. Maybe there is another way I am not aware of ! — Olivier Moindrot, Jun 06 '16 at 17:59
One question: how to solve if the installation is from binary ? — Kyrol, Aug 20 '16 at 09:11
"If using a binary install, upgrade your cudnn library to match" — Olivier Moindrot, Aug 20 '16 at 09:32
If one installs it with pip, what is the solution to this problem? Is it the same? — Charlie Parker, Oct 10 '16 at 18:04
Yes, pip install is the same as binary. You need to have the standard cudnn version — Olivier Moindrot, Oct 10 '16 at 20:05

score 1 · Answer 2 · answered Jun 06 '16 at 17:27

1

Seems like you have cudnn 5 installed. You need to set it when running ./configure

Please specify the Cudnn version you want to use. [Leave empty to use system
default]: 5

answered Jun 06 '16 at 17:27

fabrizioM

46,639
15
102
119

Can I run `./configure` in a current implementation, or do I need to recompile with Bazel? – Alexander R Johansen Jun 06 '16 at 17:34
1

@GunnarNielsen You need to recompile from the source code – fabrizioM Jun 06 '16 at 18:23
If one installs it with pip, what is the solution to this problem? Is it the same? – Charlie Parker Oct 10 '16 at 18:04

score 1 · Answer 3 · answered Feb 15 '17 at 10:38

Adding my 2 cents: In my case (TF0.12.1, installed to anaconda from pip, without sudo permissions) CuDNNv5 was installed, but was not the default.

setting export LD_LIBRARY_PATH="/usr/local/lib/cuda-8.0/lib64:/usr/local/lib/cudann5/lib64/" solved the issue

score 0 · Answer 4 · answered Sep 11 '17 at 14:07

I also meet this incompatible problem like this :

Loaded runtime CuDNN library: 5005 (compatibility version 5000) but source wascompiled with 5110 (compatibility version 5100).  If using a binary install, upgrade your CuDNNlibrary to match.  If building fromsources, make sure the library loaded at runtime matches a compatible versionspecified during compile configuration.

So i download CuDNN 5.1(compatible with CUDA8.0) and replace 5.0 with it then everything goes well.

WARN : CuDNN from nvidia is unavaliable,but you can find it from other's share.

cudnn compile configuration in TensorFlow

4 Answers4