12

I've started using docker buildx to tag and push mutli-platform images to ECR. However, ECR appears to apply the tag to the parent manifest, and leaves each related manifest as untagged. ECR does appear to prevent deletion of the child manifests, but it makes managing cleanup of orphaned untagged images complicated.

Is there a way to tag these child manifests in some way?

For example, consider this push:

docker buildx build --platform "linux/amd64,linux/arm64" --tag 1234567890.dkr.ecr.eu-west-1.amazonaws.com/my-service/my-image:1.0 --push  .

Inspecting the image:

 docker buildx imagetools inspect 1234567890.dkr.ecr.eu-west-1.amazonaws.com/my-service/my-image:1.0

Shows:

Name:      1234567890.dkr.ecr.eu-west-1.amazonaws.com/my-service/my-image:1.0
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:4221ad469d6a18abda617a0041fd7c87234ebb1a9f4ee952232a1287de73e12e
       
Manifests: 
    Name:      1234567890.dkr.ecr.eu-west-1.amazonaws.com/my-service/my-image:1.0@sha256:c1b0c04c84b025357052eb513427c8b22606445cbd2840d904613b56fa8283f3
    MediaType: application/vnd.docker.distribution.manifest.v2+json
    Platform:  linux/amd64
         
    Name:      1234567890.dkr.ecr.eu-west-1.amazonaws.com/my-service/my-image:1.0@sha256:828414cad2266836d9025e9a6af58d6bf3e6212e2095993070977909ee8aee4b
    MediaType: application/vnd.docker.distribution.manifest.v2+json
    Platform:  linux/arm64

However, ECR shows the 2 child images as untagged

Bengineer
  • 233
  • 2
  • 6
  • This seems to be running for almost 2 years at this stage (as of July 2023), related: https://github.com/aws/containers-roadmap/issues/1596 and https://github.com/aws/containers-roadmap/issues/1612, especially this comment was enlightening: https://github.com/aws/containers-roadmap/issues/1596#issuecomment-1546056529 – Drachenfels Jul 06 '23 at 23:55

3 Answers3

11

I'm running into the same problem. So far my solution seems a little easier than some of the other suggestions, but I still don't like it.

After doing the initial:

docker buildx build --platform "linux/amd64,linux/arm64" --tag 1234567890.dkr.ecr.eu-west-1.amazonaws.com/my-service/my-image:1.0 --push  .

I follow up with:

docker buildx build --platform "linux/amd64" --tag 1234567890.dkr.ecr.eu-west-1.amazonaws.com/my-service/my-image:1.0-amd --push  .
docker buildx build --platform "linux/arm64" --tag 1234567890.dkr.ecr.eu-west-1.amazonaws.com/my-service/my-image:1.0-arm --push  .

This gets me the parallel build speed of building multiple platforms at the same time, and gets me the images tagged in ECR. Thanks to having the build info cached it is pretty quick, it appears to just push the tags and that is it. In a test I just did the buildx time for the first command was 0.5 seconds. and the second one took 0.7 seconds.

That said, I'm not wild about this solution, and found this question while looking for a better one.

Bryce Wade
  • 111
  • 1
  • 2
6

There are several ways to tag the image, but they all involve pushing the platform specific manifest with the desired tag. With docker, you can pull the image, retag it, and push it, but the downside is you'll have to pull every layer.

A much faster option is to only transfer the manifest json with registry API calls. You could do this with curl, but auth becomes complicated. There are several tools for working directly with registries, including Googles crane, RedHat's skopeo, and my own regclient. Regclient includes the regctl command which would implement this like:

image=1234567890.dkr.ecr.eu-west-1.amazonaws.com/my-service/my-image
tag=1.0
regctl image copy \
  ${image}@$(regctl image digest --platform linux/amd64 $image:$tag) \
  ${image}:${tag}-linux-amd64
regctl image copy \
  ${image}@$(regctl image digest --platform linux/arm64 $image:$tag) \
  ${image}:${tag}-linux-arm64

You could also script an automated fix to this, listing all tags in the registry, pulling the manifest list for the tags that don't already have the platform, and running the image copy to retag each platform's manifest. But it's probably easier and faster to script your buildx job to include something like regctl after buildx pushes the image.

Note if you use a cred helper for logging into ECR, regctl can use this with the local command. If want to run regctl as a container, and you are specifically using ecr-login, use the alpine version of the images since they include the helper binary.

BMitch
  • 231,797
  • 42
  • 475
  • 450
  • Many thanks. This was what I was looking for. Using local docker was problematic as I couldn't pull the ARM image. Note that given we are using AWS ECR, I can also use `aws ecr batch-get-image` to get the manifest, and `aws ecr put-image` to rewrite. Your script looks more elegant however. – Bengineer Aug 16 '21 at 10:40
  • Feels like AWS should update ECR to show the image name+architecture in that column instead of . And probably show that on the image details page as well. Right now, the ECR page looks very ugly with all these names. Given that fargate only has x86 Xeon servers (no arm-based graviton servers YET), I'm guessing AWS is playing catch up on the "multiarch docker images" space. This is weird given their push of the faster/cheaper graviton... AWS: can you grant me access to your code repo so I can make the change in the UI? :) – Pierre Oct 18 '21 at 21:03
0

In addition to what Brandon mentioned above on using regctl, here's the command for skopeo if you're looking to use it with ECR credential helper. https://github.com/awslabs/amazon-ecr-credential-helper

skopeo copy \ 
docker://1234567890.dkr.ecr.us-west-2.amazonaws.com/stackoverflow@sha256:1badbc699ed4a1785295baa110a125b0cdee8d854312fe462d996452b41e7755 \
docker://1234567890.dkr.ecr.us-west-2.amazonaws.com/stackoverflow:1.0-linux-arm64

https://github.com/containers/skopeo

  • Paavan Mistry, AWS Containers DA
Paavan
  • 1