I'm trying to get the 8bit quantised bloom models running on my machine.
I (seem to have) installed all of the dependencies, however, when I run my script, it errors - saying it can't find the cudaGetRuntime
I then tried switching to to conda environemnt - using conda activate (and conda install cudatoolkit)
and it says it cant find "bitsandbytes"
root@localhost:~# pip install bitsandbytes accelerate transformers
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: bitsandbytes in /usr/local/lib/python3.10/dist-packages (0.35.4)
Requirement already satisfied: accelerate in /usr/local/lib/python3.10/dist-packages (0.15.0)
Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.25.1)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate) (21.3)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate) (5.9.4)
Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from accelerate) (5.4.1)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from accelerate) (1.23.4)
Requirement already satisfied: torch>=1.4.0 in /usr/local/lib/python3.10/dist-packages (from accelerate) (1.13.0)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.64.1)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from transformers) (2.27.1)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.8.2)
Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.13.2)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2022.10.31)
Requirement already satisfied: huggingface-hub<1.0,>=0.10.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.11.1)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.10.0->transformers) (4.4.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/lib/python3/dist-packages (from packaging>=20.0->accelerate) (3.0.7)
Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->accelerate) (11.7.99)
Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->accelerate) (11.10.3.66)
Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->accelerate) (11.7.99)
Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /usr/local/lib/python3.10/dist-packages (from torch>=1.4.0->accelerate) (8.5.0.96)
Requirement already satisfied: wheel in /usr/local/lib/python3.10/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.4.0->accelerate) (0.38.4)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.4.0->accelerate) (65.5.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@localhost:~# python /tmp/pycharm_project_354/bloom8.py
File "/tmp/pycharm_project_354/bloom8.py", line 6, in <module>
from bitsandbytes.functional import quantize_blockwise, dequantize_blockwise
ModuleNotFoundError: No module named 'bitsandbytes'
