0

I am seeing different checksums calculated for image layers that should be identical.

This is my dockerfile:

# syntax=docker/dockerfile:1.4

ARG ONEAPI_VERSION=2022.1.2
FROM intel/oneapi-hpckit:${ONEAPI_VERSION}-devel-ubuntu18.04 AS oneapi-hpckit
FROM ubuntu:18.04
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
      build-essential \
      cmake \
      ninja-build && \
    rm -rf /var/lib/apt/lists/*
COPY --link --from=oneapi-hpckit /opt/intel/oneapi/setvars.sh /opt/intel/oneapi/setvars.sh
COPY --link --from=oneapi-hpckit /opt/intel/oneapi/compiler /opt/intel/oneapi/compiler
COPY --link --from=oneapi-hpckit /opt/intel/oneapi/mkl /opt/intel/oneapi/mkl
COPY --link --from=oneapi-hpckit /opt/intel/oneapi/tbb /opt/intel/oneapi/tbb

I built it twice using:

git clone <repo> repo
docker build \
  --build-arg http_proxy \
  --build-arg https_proxy \
  --build-arg no_proxy \
  --build-arg BUILDKIT_INLINE_CACHE=1 \
  --cache-from ${DOCKER_REGISTRY}/${DOCKER_IMAGE} \
  -t ${DOCKER_REGISTRY}/${DOCKER_IMAGE} \
  - < repo/Dockerfile
docker push ${DOCKER_REGISTRY}/${DOCKER_IMAGE}
docker rmi ${DOCKER_REGISTRY}/${DOCKER_IMAGE}
docker image prune -f
rm -rf repo
git clone <repo> repo
docker build \
  --build-arg http_proxy \
  --build-arg https_proxy \
  --build-arg no_proxy \
  --build-arg BUILDKIT_INLINE_CACHE=1 \
  --cache-from ${DOCKER_REGISTRY}/${DOCKER_IMAGE} \
  -t ${DOCKER_REGISTRY}/${DOCKER_IMAGE} \
  - < repo/Dockerfile

In both instances all of the stages were cached, no stage was rebuilt.

These build steps have matching hashes:

#16 [stage-1 1/6] FROM docker.io/library/ubuntu:18.04@sha256:478caf1bec1afd54a58435ec681c8755883b7eb843a8630091890130b15a79af
#16 sha256:f6e80c6e585e12e84ea064dc5bfbdde0762abb0f45c119d1c3c6b06ccf3c31db
#16 DONE 0.0s

#10 [stage-1 1/6] FROM docker.io/library/ubuntu:18.04@sha256:478caf1bec1afd54a58435ec681c8755883b7eb843a8630091890130b15a79af
#10 sha256:f6e80c6e585e12e84ea064dc5bfbdde0762abb0f45c119d1c3c6b06ccf3c31db
#10 DONE 0.0s

#11 [oneapi-hpckit 1/1] FROM docker.io/intel/oneapi-hpckit:2022.1.2-devel-ubuntu18.04@sha256:f87d873d7de912c9f7d944cea1da057545d10fdd63970afb4389bc9f02037408
#11 sha256:555017ddeca0a3576363999f93ef738a8692e02ade19f1f177b9578f4f524c93
#11 DONE 0.0s

#12 [oneapi-hpckit 1/1] FROM docker.io/intel/oneapi-hpckit:2022.1.2-devel-ubuntu18.04@sha256:f87d873d7de912c9f7d944cea1da057545d10fdd63970afb4389bc9f02037408
#12 sha256:555017ddeca0a3576363999f93ef738a8692e02ade19f1f177b9578f4f524c93
#12 DONE 0.0s

These build steps have mismatching hashes:

#10 [stage-1 2/6] RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y       build-essential       cmake       ninja-build &&     rm -rf /var/lib/apt/lists/*
#10 sha256:7573e91567e9bfe53570e6ea939f129389b7e45feb076817817f751fab5619ab
#10 CACHED

#11 [stage-1 2/6] RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y       build-essential       cmake       ninja-build &&     rm -rf /var/lib/apt/lists/*
#11 sha256:ff984e2077208e0fe2c6312adb6ff8203fe87e722082b5dbc41c341c812e2624
#11 CACHED

#12 [stage-1 3/6] COPY --link --from=oneapi-hpckit /opt/intel/oneapi/setvars.sh /opt/intel/oneapi/setvars.sh
#12 sha256:393028c2915e599f51b32f595eacdea522dfcd5abe118df11e59aef593b13485
#12 CACHED

#13 [stage-1 3/6] COPY --link --from=oneapi-hpckit /opt/intel/oneapi/setvars.sh /opt/intel/oneapi/setvars.sh
#13 sha256:c56ddffb0433f94dbfcebe0858dc729c7bda25a342f95416d1651ba45618f05e
#13 CACHED

#13 [stage-1 4/6] COPY --link --from=oneapi-hpckit /opt/intel/oneapi/compiler /opt/intel/oneapi/compiler
#13 sha256:d4b8865ff314458f268cb8bb4d0f049ebad5537fb3cf8b45802b9f63a44da007
#13 CACHED

#14 [stage-1 4/6] COPY --link --from=oneapi-hpckit /opt/intel/oneapi/compiler /opt/intel/oneapi/compiler
#14 sha256:cba2dff7420415fa1cebc119bf845726143ba56af4431827b561dd012176a2fd
#14 CACHED

#14 [stage-1 5/6] COPY --link --from=oneapi-hpckit /opt/intel/oneapi/mkl /opt/intel/oneapi/mkl
#14 sha256:a7ef5221a3a6a98c6191d2afa211e91a23bb23073973ac5bdb2404806714aeb1
#14 CACHED

#15 [stage-1 5/6] COPY --link --from=oneapi-hpckit /opt/intel/oneapi/mkl /opt/intel/oneapi/mkl
#15 sha256:8d367ab53ddf708a0ce984aabee31f2ab6f9543b6fa8475d6f12db7bcfe16076
#15 CACHED

#15 [stage-1 6/6] COPY --link --from=oneapi-hpckit /opt/intel/oneapi/tbb /opt/intel/oneapi/tbb
#15 sha256:1fa326be1295455c08fe072fb888b8c49cbde3f160161107eb237cb8d638ac10
#15 CACHED

#16 [stage-1 6/6] COPY --link --from=oneapi-hpckit /opt/intel/oneapi/tbb /opt/intel/oneapi/tbb
#16 sha256:cd87f15a1299ebac4a0bc24f6d4434b74ff54c20ef85513747a8cb9b5b6dbb01
#16 CACHED

So the hashes deviate at stage-1 2/6, but in both cases they are cached from the remote registry (so not rebuilt), and stage-1 1/6 hashes are identical. Nothing beside that push updated the image in the registry in between these builds.

The imported cache manifest from the registry is the same for both builds (sha256:0cf935a9ac4a10c138f2f7d52a92c850f55aa56827a794edb72997f628250f71), and the metadata for both base images are also the same. The only difference I see between the build logs is the hashes for [internal] load build definition from Dockerfile, but I guarantee that the Dockerfile did not change between runs. The metadata may have changed on it, since each build was from a different git checkout, but I guarantee the file content is identical between checkouts.

How did these hashes deviate?

Lisanna
  • 133
  • 1
  • 4

1 Answers1

0

I discovered that it was a case of the local build cache having a different version of the image layers than the registry (in the --cache-from, and one was chosen at random each time (the image layers in the registry also existed in the local build cache). Before clearing the build cache on the machine with docker builder prune -f, which of the two layers that was used was random. After clearing the build cache, the same layer was used every time (from the registry).

Lisanna
  • 133
  • 1
  • 4