CuDNN is not available in binary-installed cupy

Question

I installed Cupy using the binary package cupy-cuda92, but CuDNN is not loaded.

$ pip install cupy-cuda92==5.4.0 chainer==5.4.0

$ python -c 'import chainer; chainer.print_runtime_info()'
/home/xxxxx/.pyenv/versions/3.5.2/lib/python3.5/site-packages/chainer/backends/cuda.py:98: UserWarning: cuDNN is not enabled.
Please reinstall CuPy after you install cudnn
(see https://docs-cupy.chainer.org/en/stable/install.html#install-cudnn).
  'cuDNN is not enabled.\n'
Platform: Linux-4.4.0-103-generic-x86_64-with-debian-stretch-sid
Chainer: 5.3.0
NumPy: 1.16.2
CuPy:
  CuPy Version          : 5.3.0
  CUDA Root             : /usr/local/cuda
  CUDA Build Version    : 9020
  CUDA Driver Version   : 9020
  CUDA Runtime Version  : 9020
  cuDNN Build Version   : None
  cuDNN Version         : None
  NCCL Build Version    : 2307
  NCCL Runtime Version  : 2402
iDeep: Not Available

I tried uninstalling & reinstalling chainer and cupy with --no-cache-dir but saw no difference. I also confirmed that CUDA is 9.2.

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Wed_Apr_11_23:16:29_CDT_2018
Cuda compilation tools, release 9.2, V9.2.88

score 2 · Answer 1 · answered Apr 24 '19 at 05:24

I found the problem and a solution on my own.

First I tried to import cudnn directly.

$ python
Python 3.5.2 (default, Mar 25 2019, 10:54:56)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cupy.cuda.cudnn
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /home/xxxxx/.pyenv/versions/3.5.2/lib/python3.5/site-packages/cupy/cuda/cudnn.cpython-35m-x86_64-linux-gnu.so: symbol cudnnGetBatchNormalizationTrainingExReserveSpaceSize, version libcudnn.so.7 not defined in file libcudnn.so.7 with link time reference

thus there is a kind of library version mismatch. I checked my LD_LIBRARY_PATH and found that different versions of CuDNN are installed via cudnnenv. I removed them from LD_LIBRARY_PATH and the problem was gone.

$ python -c 'import chainer; chainer.print_runtime_info()'
Platform: Linux-4.4.0-103-generic-x86_64-with-debian-stretch-sid
Chainer: 5.3.0
NumPy: 1.16.2
CuPy:
  CuPy Version          : 5.3.0
  CUDA Root             : /usr/local/cuda
  CUDA Build Version    : 9020
  CUDA Driver Version   : 9020
  CUDA Runtime Version  : 9020
  cuDNN Build Version   : 7402
  cuDNN Version         : 7500
  NCCL Build Version    : 2307
  NCCL Runtime Version  : 2402
iDeep: Not Available

FYI: `LD_LIBRARY_PATH` will be ignored in CuPy v6 (always use cuDNN bundled in the binary package), which is planned to be released in the next month. https://github.com/cupy/cupy/pull/1770 — kmaehashi, Apr 25 '19 at 01:29

CuDNN is not available in binary-installed cupy

1 Answers1