Docker how to run pip requirements.txt only if there was a change?

Question

In a Dockerfile I have a layer which installs requirements.txt:

FROM python:2.7
RUN pip install -r requirements.txt

When I build the docker image it runs the whole process regardless of any changes made to this file.

How do I make sure Docker only runs pip install -r requirements.txt if there has been a change to the file?

Removing intermediate container f98c845d0f05
Step 3 : RUN pip install -r requirements.txt
 ---> Running in 8ceb63abaef6
Collecting https://github.com/tomchristie/django-rest-framework/archive/master.zip (from -r requirements.txt (line 30))
  Downloading https://github.com/tomchristie/django-rest-framework/archive/master.zip
Collecting Django==1.8.7 (from -r requirements.txt (line 1))

Answer 1

I'm assuming that at some point in your build process, you're copying your entire application into the Docker image with COPY or ADD:

COPY . /opt/app
WORKDIR /opt/app
RUN pip install -r requirements.txt

The problem is that you're invalidating the Docker build cache every time you're copying the entire application into the image. This will also invalidate the cache for all subsequent build steps.

To prevent this, I'd suggest copying only the requirements.txt file in a separate build step before adding the entire application into the image:

COPY requirements.txt /opt/app/requirements.txt
WORKDIR /opt/app
RUN pip install -r requirements.txt
COPY . /opt/app
# continue as before...

As the requirements file itself probably changes only rarely, you'll be able to use the cached layers up until the point that you add your application code into the image.

Answer 2

This is directly mentioned in Docker's own "Best practices for writing Dockerfiles":

If you have multiple Dockerfile steps that use different files from your context, COPY them individually, rather than all at once. This will ensure that each step’s build cache is only invalidated (forcing the step to be re-run) if the specifically required files change.

For example:

COPY requirements.txt /tmp/
RUN pip install --requirement /tmp/requirements.txt
COPY . /tmp/

Results in fewer cache invalidations for the RUN step, than if you put the COPY . /tmp/ before it.

Answer 3

Alternatively as a quicker means to run requirements.txt file without typing "yes" to confirm installation of libraries, you can re-write as:

COPY requirements.txt ./
RUN pip install -y -r requirements.txt
COPY ./"dir"/* .