0

I'm trying to use/test the (experimental) AMD GPU support of CuPy (8.1.0). In short: I'm having an ImportError.

  • I followed the instructions for installing ROCm on a Ubuntu focal machine (with a gfx906/Radeon VII (Vega 20) card).
  • From the doc I set the mentioned variables (with adjusted HCC_AMDGPU_TARGET), the ROCM_HOME and export PATH=$PATH:/opt/rocm/bin:/opt/rocm/profiler/bin:/opt/rocm/opencl/bin.
  • pip3 install -v --no-cache-dir cupy succeeded without any complaints. Just a few compiler notes and warnings because of the -v option
  • Now when I try to import cupy it's throwing an ImportError.
In [1]: import cupy                                                                                                                                                                                                                         
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/usr/local/lib/python3.8/dist-packages/cupy/__init__.py in <module>
     19             message='can\'t resolve package from __spec__')
---> 20         from cupy import core  # NOQA
     21 except ImportError as e:

/usr/local/lib/python3.8/dist-packages/cupy/core/__init__.py in <module>
----> 1 from cupy.core import core  # NOQA
      2 from cupy.core import internal  # NOQA
      3 

cupy/core/core.pyx in init cupy.core.core()
cupy/core/_routines_manipulation.pyx in init cupy.core._routines_manipulation()
cupy/core/_routines_indexing.pyx in init cupy.core._routines_indexing()
cupy/core/_routines_math.pyx in init cupy.core._routines_math()
cupy/core/_reduction.pyx in init cupy.core._reduction()
cupy/core/_cub_reduction.pyx in init cupy.core._cub_reduction()

ModuleNotFoundError: No module named 'cupy.cuda.cub'

The above exception was the direct cause of the following exception:

ImportError                               Traceback (most recent call last)
<ipython-input-1-329ec5cf1bc8> in <module>
----> 1 import cupy

/usr/local/lib/python3.8/dist-packages/cupy/__init__.py in <module>
     39 original error: {}'''.format(_exc_info[1]))  # NOQA
     40 
---> 41     raise ImportError(_msg) from e
     42 
     43 

ImportError: CuPy is not correctly installed.
If you are using wheel distribution (cupy-cudaXX), make sure that the version of CuPy you installed matches with the version of CUDA on your host.
Also, confirm that only one CuPy package is installed:
  $ pip freeze

If you are building CuPy from source, please check your environment, uninstall CuPy and reinstall it with:
  $ pip install cupy --no-cache-dir -vvvv

Check the Installation Guide for details:
  https://docs.cupy.dev/en/latest/install.html

original error: No module named 'cupy.cuda.cub'

What should I do next? What should I look for in the compilation process? Thanks in advance

EDIT On request here the complete stdout and stderr of the installation process and a few io's of the console: https://seafile.cloud.uni-hannover.de/d/68862cd1057e47d180aa/

Daniel
  • 141
  • 1
  • 10
  • Do you have `hipcub` and `rocprim` installed before building CuPy? They are required dependencies. – Leo Fang Oct 30 '20 at 19:36
  • Yes. I `apt-get install`ed the complete line from the doc. And later `export`ed the variables and executed the `pip3` line. All as "user" root (I have my reasons why (here) I don't use sudo) – Daniel Oct 30 '20 at 20:31
  • It'd be nice if you could find a way to post the full install log here (output of `pip3 install -v ...`). Another thing to try is to clone the v8 branch and then build from source `git clone -b v8.1.0 https://github.com/cupy/cupy.git; cd cupy; pip3 install -v .`), just to see if it's an issue with the sdist on PyPI. – Leo Fang Oct 31 '20 at 00:36
  • (btw I am the main driver behind CuPy's ROCm support these days) – Leo Fang Oct 31 '20 at 00:37
  • @LeoFang I uploaded the requested logs to our seafile folder. I'll try your git approach later – Daniel Nov 02 '20 at 17:35
  • Thanks. I think the reason is in CuPy v8 we assumed the `cupy.cuda.cub` module is always built, but the ROCm support was not added until recently. I think this can be fixed in the next stable release (v8.2). In the meanwhile, please build from the master branch (currently v9.0a1). It has more extensive coverage over ROCm libraries. – Leo Fang Nov 02 '20 at 19:53
  • btw I opened an issue: https://github.com/cupy/cupy/issues/4218 – Leo Fang Nov 02 '20 at 19:57
  • So first pip tells me that the folder `cupy/core/include/cupy/cub` is a "git submodule but is currently empty". And after populating the subfolder with git and `pip3 install .` an exception is raised by `install/build.py` line 196 in `get_compiler_setting`: "Please install hipCUB and retry". But the `hipcub` package is installed and things like `hipcc` are in the PATH – Daniel Nov 03 '20 at 15:33
  • Could you make sure your `hipcub` and `rocprim` are really installed? If so, you should be able to find folders locally like `/opt/rocm/include/hipcub`, `/opt/rocm-3.5.0/include/hipcub`, `/opt/rocm/hipcub/`, `/opt/rocm-3.5.0/hipcub/`, etc – Leo Fang Nov 03 '20 at 21:23
  • Strange, but indeed the mentioned include/hipcub did not existed. Meanwhile I'm facing another problem unrelated to my question above. I cannot reinstall `hsakmt-roct` and `rocsolver` because of an unexpected size mismatch, i.e. 74018 != 74022 @hsakmt-roct. For some reason I need to do this because rocminfo complains about a missing libhsakmt.so.1. I'll work on that later – Daniel Nov 13 '20 at 14:23
  • I think it's likely because ROCm evolves too fast, and if you install from their repo through your package manager (ex: `sudo apt instal ...`) it's likely to pull incompatible versions (say, some on 3.8.0 and some on 3.9.0) and lead to the issues your observed. You likely need to do a clean install, and either follow their multi-versioning scheme or ensure a single version is installed. – Leo Fang Nov 15 '20 at 20:23

1 Answers1

0

To summarize the discussion in the comments so far:

  • For CuPy v8.x (the latest stable releases): Up to v8.1 the offending cupy.cuda.cub module is not built in ROCm/HIP environments, which will hopefully be fixed in v8.2 (see ticket).
  • For CuPy v9.x (the master branch): It should just work as long as rocPRIM and hipCUB are correctly installed. It's likely that the OP's local environment is not correctly set up, leading to cupy.cuda.cub module not built and thus ImportError.
Leo Fang
  • 773
  • 5
  • 12