AWS Digest always changes on tagging

Question

I'm currently creating a python script in order to create an adapter to AWS ECR using AWS CLI. I have a staging environment with a versioned test container in version 1.0.0. Now I try to retag it using the following control flow:

Save the manifest.json of the versioned container: aws ecr batch-get-image --repository-name staging/test --image-ids imageTag=1.0.0 --output text (the json output of the aws cli is buggy and cannot be parsed in a robust way)
Delete the old "latest" tag: aws ecr batch-delete-image --repository-name staging/test --image-ids imageTag=latest
Use the image digest and the manifest json to re-tag the image: aws ecr put-image --repository-name staging/test --image-tag latest --image-manifest file://manifest.json --image-digest sha256:foobar

The problem: The ECR throws an exception describing that the image digest does differ. But there is no 'latest' image - it was untagged. So I tried to tag the image to a non-existing version '2.0.0' and one time without the digest. The ECR tags the image and calculates a new image digest - which seems to be the core problem. Sense - it does not make any.

Same question here https://stackoverflow.com/q/64686471/11715259 — N1ngu, Jul 21 '23 at 07:32

score 1 · Answer 1 · answered Jun 07 '22 at 05:57

1

After contacting the AWS support it seems that the json changes once you use plain text output of the cli. If you use the "jq" provided solution of AWS, everything works fine. Other ways will alter the whitespaces and the the ECR will calculate a different digest.

In the end it seems odd, that the content of the manifest is as important as the sorrounding json syntax with whitespaces.

answered Jun 07 '22 at 05:57

Mr.Mountain

863
8
32

I kinda think AWS recently changed how it does the digest calculation, because I'm seeing operations like this result in different digests and a couple days ago that was not happening: `docker pull ${ACCOUNT}.dkr.ecr.us-west-2.amazonaws.com/${IMAGE}:tag1; docker tag ${ACCOUNT}.dkr.ecr.us-west-2.amazonaws.com/${IMAGE}:tag1 ${ACCOUNT}.dkr.ecr.us-west-2.amazonaws.com/${IMAGE}:tag2; docker push ${ACCOUNT}.dkr.ecr.us-west-2.amazonaws.com/${IMAGE}:tag2` – casey gerstle Jul 19 '22 at 20:36

score 1 · Answer 2 · answered May 09 '23 at 10:47

So, it turns out the image digest of an image is the sha256 digest of its manifest.

This explains why different whitespace in the manifest JSON file leads to different digests.

A strange consequence is that, while the manifest returned by aws ecr batch-get-image --query 'images[0].imageManifest' is exactly the document you need, piping it to a file will easily add a final newline character that modifies the manifest digest.

A solution is to trim that last character with head -c -1

aws ecr batch-get-image --repository-name staging/test --image-ids imageTag=1.0.0 --output=text --query 'images[0].imageManifest' | head -c -1 > manifest.json

aws ecr put-image --repository-name staging/test --image-tag latest --image-manifest file://manifest.json --image-digest sha256:foobar

score 0 · Answer 3 · answered Aug 28 '23 at 16:35

I fixed this (after much frustration with the following python code):

    # read exact "manifest" of src_tag to be duplicated
    _, img_data = run_cmd(
        f"aws ecr batch-get-image --repository-name {repo_name} --image-ids imageTag={src_tag} --output json --region {AWS_CONF['region']} --profile {AWS_CONF['profile']}",
    )
    img_data = json.loads(img_data)
    manifest_str = img_data["images"][0]["imageManifest"]

    # retag image (using boto3 instead of CLI to ensure the exact manifest string is passed in)
    import boto3

    ecr_client = boto3.client("ecr", region_name=AWS_CONF["region"])
    response = ecr_client.put_image(
        registryId=registry_id,
        repositoryName=repo_name,
        imageManifest=manifest_str,
        imageManifestMediaType="string",
        imageTag=dest_tag,
    )
    if response["ResponseMetadata"]["HTTPStatusCode"] != 200:
        print(f"ERROR: failed to retag image! (status code 200)")

Note that run_cmd() is a custom function that calls a subprocess and returns the output as a string. Note that reading the manifest of the src_tag could almost certainly be done with the boto3 API instead but I chose not to refactor my code further.

For me I was unable to get this to work with aws ecr batch-get-image because the manifest string would always slightly change. This solution ensures the exact same manifest string as the src_tag is used (avoiding modifying it indirectly by parsing it as json etc).

AWS Digest always changes on tagging

3 Answers3

Linked