5

I'm building Docker containers in a GitHub Actions workflow, and I have a custom caching solution that uses Docker buildx. Essentially, I use the docker buildx build --cache-from and --cache-to arguments to dump the cache from the last build, store it externally, and load it for the next built in order to save time.

For the most part, this is working beautifully. Most of the build time is spent on pip install since we're using Alpine and there are no wheels available, so it has to build the packages, so skipping that step with a cache is a huge savings.

There are two relevant lines in the Dockerfile:

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

I'm encountering this odd issue, though, where requirements.txt doesn't change, and the first line (the COPY) correctly uses the cache, but the second line (the RUN) does not use the cache, and proceeds to re-build all of the packages.

I cannot figure out what could be causing this. If the COPY uses the cache properly, then it must be recognizing that requirements.txt hasn't changed, and if requirements.txt hasn't changed, I don't understand why it wouldn't use the cache for the RUN command.

And the worst part is, it's intermittent. Some builds it uses the cache, some builds it re-builds the packages, when in neither case has requirements.txt (or any preceeding Dockerfile line) changed.

Am I missing something?

Jordan
  • 3,998
  • 9
  • 45
  • 81
  • In your _requirements.txt_ do you pin version to an exact version number? – β.εηοιτ.βε Dec 28 '21 at 23:07
  • I do, but even if I didn't, Docker should still use the cached layer since Docker has no way of knowing if there have been any updates to the underlying packages. – Jordan Dec 29 '21 at 01:02
  • 1
    Found a similar [discussion](https://github.com/docker/buildx/issues/402). Can you try with `mode=max` on your `cache-to` arg? – Manish Dash Apr 14 '22 at 10:16
  • @ManishDash I am already using `mode-max`, unfortunately. – Jordan Apr 14 '22 at 23:18
  • if you want to use cache, why you disable it using `pip`? (e.g. why adding the `--no-cache-dir` flag)? feels like I'm missing something here – ymz Apr 19 '22 at 11:36
  • 1
    `--no-cache-dir` is probably to avoid pip creating new (cache) files during build which won't be used and would only cause image to be bigger, rather than something related to using existing files – Pierre B. Apr 19 '22 at 15:44
  • Exactly, the point is to cache the layer with Docker cache, not the packages with pip cache. – Jordan Apr 19 '22 at 19:13

0 Answers0