0

The Dockerfile contains:

RUN /bin/bash -c "python3 -m pip install --upgrade pip && python3 -m pip install conan"

Once built, it never run it again and use the cache instead. I'm fine as long as the versions (pip's + conan's) haven't changed.

What's the best practice to handle that case? I'd like docker/buildah to detect whether it needs to change the layer if there is a new version. Purposely, I didn't add any version to always get the latest versions.

I struggled finding the cause of a bug I had, Conan has changed their SSL certificate in a new version and I was stuck with a previous version that prevented me from installing packages.

Alexis
  • 2,136
  • 2
  • 19
  • 47

1 Answers1

0

This conflict is exactly why to use a version lock file that lists out specific versions of packages to use. Even if you think you usually want the latest version, this gives you (or the packaging tool) a place to record a specific set of versions that you know works.

Using Python's basic setuptools system, for example, you can declare in your setup.cfg file that your application needs that specific package

[options]
install_requires=
    conan

Now in your local (non-Docker) development environment, you can install that

rm -rf venv           # clean up the old virtual environment
python3 -m venv venv  # create a new virtual environment
. venv/bin/activate   # activate it
pip install -e .      # install current directory and its dependencies

or alternatively, if you already have the virtual environment set up

pip install --upgrade -e .

Now you can ask pip to dump out the requirements file

pip freeze > requirements.txt

In your Dockerfile, COPY the new requirements.txt file in, and use that to install packages.

FROM python:3.9
# Upgrade pip, since it likes to complain about being out of date
RUN pip install --upgrade pip
# Install package dependencies
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
# Install the rest of the application
COPY . .
CMD ["./application.py"]

Docker's layer caching system means, if the requirements.txt file has changed, Docker will re-run pip install, and if it hasn't, it will skip over it (until the next COPY line with a changed file). Meanwhile, it's under your control which exact version to use (you can either put a version constraint in setup.cfg or manually edit requirements.txt to avoid a broken version) so a breaking upstream change won't mean you can't ship code. Finally, you're using the same packaging system in your development environment and in Docker, so it's easy to keep things consistent (I'd generally discourage pip installing individual packages in a Dockerfile).

David Maze
  • 130,717
  • 29
  • 175
  • 215
  • Thanks but that solves the opposite problem, when Docker creates new layer whereas the version hasn't changed. (found that solution on tens of websites already) – Alexis Nov 26 '21 at 12:49
  • I want to get a new layer when there is a new version, I'm not willing to define static versions, I always want the latest version. The line `RUN pip install --upgrade pip` doesn't trigger a new layer even though there is a new version. That's my issue, not the opposite. I wonder why Docker doesn't think there is a change for commands like install or update. True for yum, apt etc... – Alexis Nov 26 '21 at 12:51
  • Docker's caching system _only_ looks at your local source tree. It has no way to know if something upstream has changed for `apt-get`, `apk`, `pip`, `yum`, `git`, `hg`, `svn`, `curl`, `wget`, or any other command that reaches "outside" the local system. – David Maze Nov 26 '21 at 14:03
  • 1
    You can use `docker build --no-cache` without a lock file; but it will be much slower (because Docker needs to repeat the installation on every build) and your build will occasionally break (because upstream made an incompatible change or had a bug). – David Maze Nov 26 '21 at 14:05
  • Yes that's what I use but it rebuilds everything not just that command. I mainly use downstream (centos) and it isn't that important if it breaks. It's strange docker doesn't have a mechanism to give the user the control over it per command. Even without downloading and installing, we can think of controlling it with which value a script/command returns compared to previous cached value. Anyway, thank you for your help! – Alexis Nov 26 '21 at 14:28
  • Simple example: Wanna drop everything from all the interfaces, permanent and temporary but not from an interface created by a VPN (wireguard). Good luck to work with that without changing both chains. Your VPN client's chain MUST take into account the existing chains/rules... – Alexis Nov 28 '21 at 14:07