1

I'm trying to set up a Docker container (RHEL8) with Kaniko. IN the Dockerfile I specified to install Python3.8 and PIP3 to install Python libraries that were requested for the specific container. requirements.txt lists about 9 libraries (joblib, nltk, numpy, pandas, scikit-learn, scipy, spacy, torch, transformers), from some of which are quite large in size (for example Torch: 890M) but then, when I run

RUN python3.8 -m pip install -r requirements.txt

it runs through the requirements.txt from top to bottom, downloads them but then after the last line it also downloads a lot of other folders/packages too, some quite huge in size, like:

nvidia-cublas-cu11 : 317M
nvidia-cudnn-cu11 : 557M

It installs a lot of packages, like: MarkupSafe, blis, catalogue, certifi, charset-normalizer, click, confection, cymem, filelock, huggingface-hub, idna, jinja, langcodes, murmurash, etc.. and the list is quite impressive

I had to increase the disk size of the runner with 6G in order to even cope with the increased amount of downloaded stuff, but the build still fails upon Taking a snapshot of the full filesystem, due to running out of free disk space.

I have increased free disk space from 8G to 10G and then as a second attempt, to 14G, but the build still fails. I have also specified --single-snapshot option for Kaniko to only take one single snapshot at the end, and not creating separate snapshots at every step (RUN, COPY). I have installed an Nvidia driver to the container, for which I picked a quite lightweight one (450.102.04) which should not take up too much space either.

My question is: are the packages installed by pip3 after installing the list specified in requirements.txt basically dependencies, that I still must install, or are those optional?

Is there any option to overcome this excessing disk space issue? When I start the build process (via GitLab CI - Kaniko) the available free space on the xfs is 12G from 14G, so I should be enough, but the build fails with exit code 1 and message: "no space left on drive"

vvvvv
  • 25,404
  • 19
  • 49
  • 81
Nysa-522
  • 49
  • 3

2 Answers2

1

I got into the exact same problem you did, where my RUN pip install -r requirements.txt --no-cache-dir stage would simply not finish. Similarly to you, I was also including a torch package in my requirements.txt and even after installation of all the packages, new ones would keep on popping up and installing including these ones you have stated:

nvidia-cublas-cu11 : 317M
nvidia-cudnn-cu11 : 557M

I was able to solve the issue by removing the torch package from my requirement.txt file and installing it manually with another RUN pip install command in the dockerfile. Here is my updated dockerfile:

FROM python:3.8

WORKDIR /app

COPY requirement.txt requirement.txt
RUN pip3 install -r requirement.txt --no-cache-dir
RUN pip3 install https://download.pytorch.org/whl/cpu/torch-1.7.1%2Bcpu-cp38-cp38-linux_x86_64.whl
COPY . .

EXPOSE 5000
CMD [ "python", "app.py"]

To my relief, the docker build I ran using this dockerfile did not take longer than 5 minutes, with my requirements.txt containing around 40 packages. I am hardly an expert, and I suppose that once the initial set of packages were installed by requirements.txt, then there were no complications (such as chain of dependencies etc) and the next pip install for torch ran seamlessly. Hoping someone else could better explain this. Oh, and I used a CPU version of torch as you may have noticed in the dockerfile, which may have also led to those 2 packages mentioned above not installing. Hope it helps.

AhmedBilalMalik
  • 36
  • 2
0

Are the packages installed by pip3 after installing the list specified in requirements.txt basically dependencies, that I still must install, or are those optional?

If they are installed, it means that a package requires them. It could be that given your use of the package, you would not need the installed dependency, but that is not guaranteed. To check that, install the environment on your own computer (everything!) and then

  1. pip uninstall <package_name>
  2. Try to run your code and see if it fails
  3. If it fails, you need package_name, if not, you don't need it and you can add this pip uninstall <package_name> -y inside the Dockerfile.

Note that this is very dangerous and I do not recommend it. If you change one call in your scripts, you could need a dependency you did not need previously and your script would fail and you would have to start the process all over.

It leaves a very fragile / clunky installation that could fail at any change you make in the packages, your scripts etc.

Is there any option to overcome this excessing disk space issue?

There is a simple way to save some space: pip uses caching to avoid re-downloading packages if need be in the future. Since you make only one install, you can disable the cache by running pip like this:

RUN python3.8 -m pip install -r requirements.txt --no-cache-dir

I just tried with pip install torch and it would have saved around 500M of disk usage for this pip install only.

vvvvv
  • 25,404
  • 19
  • 49
  • 81