How do I push helm charts to Amazon ECR

Question

I have a helm chart that I want to push to ECR but I keep hitting 401 Unauthorized.

$ helm chart list
REF                                         NAME                    VERSION     DIGEST  SIZE        CREATED 
myecr.us-east-2.amazonaws.com/hello-world   hello-world             1.0.0+1     6c7c951 135.3 KiB   23 hours

$ helm chart push myecr.us-east-2.amazonaws.com/hello-world:1.0.0
The push refers to repository [1myecr.us-east-2.amazonaws.com/helloworld]
ref:     myecr.us-east-2.amazonaws.com/hello-world:1.0.0
digest:  6c7c9512d309b04816afd17dcdaaa64d0492550d8e290155973ddab125815da7
size:    135.3 KiB
name:    hello-world
version: 1.0.0+1
Error: unexpected response: 401 Unauthorized

I also tried authenticating the ECR with helm with helm registry login myecr.us-east-2.amazonaws.com but the credentials that I got from aws sts get-caller-identity does not work.

$ aws sts get-caller-identity
{
    "UserId": "<USERID>",
    "Account": "<Account>",
    "Arn": "arn:aws:iam::<Account>:user/foo"
}

$ helm registry login myecr.us-east-2.amazonaws.com
Username: <USERID>
Password: 
Error: login attempt to https://myecr.us-east-2.amazonaws.com/v2/ failed with status: 401 Unauthorized

My helm version is v3.0.2. Does helm not support ECR as a registry for charts?

Answer 1

I found the answer so I'm answering my own question.

To authenticate helm with ECR, run:

TOKEN=`aws ecr get-login --region ${REGION} --registry-ids ${ACCOUNT} | cut -d' ' -f6`

helm registry login myecr.us-east-2.amazonaws.com
Username: AWS
Password: $TOKEN

The above will authenticate helm with ECR, however, looks like ECR doesn't support ORAS (OCI Registry As Storage). In other words, you cannot push helm charts to it just yet.

$ helm chart push myecr.us-east-2.amazonaws.com/hello-world:1.0.0
The push refers to repository [myecr.us-east-2.amazonaws.com/hello-world]
ref:     myecr.us-east-2.amazonaws.com/hello-world:2.0.0
digest:  6c7c9512d309b04816afd17dcdaaa64d0492550d8e290155973ddab125815da7
size:    135.3 KiB
name:    hello-world
version: 1.0.0+1
Error: failed commit on ref "manifest-sha256:262e1e34f4762606ec011c776944636c003969a2cfb289776fa0f7c26883f7ad": unexpected status: 405 Method Not Allowed

The issue is tracked here: https://github.com/aws/containers-roadmap/issues/308

Update: ECR support for helm chart is live https://docs.aws.amazon.com/AmazonECR/latest/userguide/push-oci-artifact.html

Answer 2

The safest and most secure with AWS CLI 2 is to pipe the token returned by aws get-login-password to helm registry login, this way the token is not stored anywhere ever. You may also need to set HELM_EXPERIMENTAL_OCI=1 in your environment to use this feature:

$ aws --version
aws-cli/2.0.41 Python/3.7.3 ...
$ ECR_HOST="<ACCOUNT>.dkr.ecr.<REGION>.amazonaws.com"
$ export HELM_EXPERIMENTAL_OCI=1
$ aws ecr get-login-password | helm registry login --username AWS --password-stdin $ECR_HOST
Login succeeded

Now you can push a chart from local registry. First save it to local:

$ helm chart save path/to/your-chart $ECR_HOST/your-chart:VERSION
ref:     <ECR_HOST>/your-chart:1.0
digest:  466005961...
size:    2.1 KiB
name:    your-chart
version: <Chart.yml "version">
1.0: saved

$ helm chart list
REF                     NAME        VERSION             DIGEST  SIZE    CREATED   
<ECR_HOST>/your-cha...  your-chart  2020.1.1-abc1234    4660059 2.1 KiB 40 seconds

Then push it to ECR:

$ helm chart push $ECR_HOST/your-chart:VERSION
The push refers to repository [<ECR_HOST>/your-chart]
ref:     <ECR_HOST>/your-chart:VERSION
digest:  46600596...
size:    2.1 KiB
name:    your-chart
version: 2020.1.1-abc1234
1.0: pushed to remote (1 layer, 2.1 KiB total)

Proof that it is there:

$ helm chart list
REF                     NAME        VERSION             DIGEST  SIZE    CREATED   
<ECR_HOST>/your-char... your-chart  2020.1.1-abc1234    4660059 2.1 KiB 11 minutes
$ helm chart remove <ECR_HOST>/your-chart:1.0
1.0: removed

$ helm chart list
REF NAME    VERSION DIGEST  SIZE    CREATED

$ helm chart pull $ECR_HOST/your-chart:1.0
1.0: Pulling from <ECR_HOST>/your-chart
ref:     <ECR_HOST>/your-chart:1.0
digest:  4660059618c...
size:    2.1 KiB
name:    your-chart
version: 2020.1.1-abc1234
Status: Downloaded newer chart for <ECR_HOST>/your-chart:1.0

$ helm chart list
REF                     NAME        VERSION             DIGEST  SIZE    CREATED   
<ECR_HOST>/your-char... your-chart  2020.1.1-abc1234    4660059 2.1 KiB 13 minutes

However there are several things that will delay us adopting the ECR as a helm charts repo:

• helm chart list clips chart ref with ellipses when past a certain length and there doesn't seem to be a way to change this (helm 3.2.1), very annoying because it clips the VERSION portion which you need for other operations
• if you look in the ECR repo for your-chart, you will see an image with tag VERSION and there is no indication whatever that it is not a docker image but a chart; it looks like TAG is arbitrary so you could have TAG be "chart-VERSION" but this feels like a hack
• the correspondence between the chart version (from the Chart.yml) and the VERSION when you save the chart is unclear; in fact when you look in the ECR repo for your-chart, you do not see the version string that is inside the Chart.yml, you just see the tag you used when you saved the chart

All these make ECR-repo-as-helm-chart-repo feel unintuitive and early stage and not in fact ready for mainstream use in a team.

Answer 3

In my case I was facing the below issue, it has nothing to do with helm registry login, nor helm-chart itself

Error: failed to do request: Post "https://3594XXXXX.dkr.ecr.us-west-2.amazonaws.com/v2/my-project/my-team/helm-charts/blobs/uploads/": EOF

The problem is with the repository name, I think AWS ECR expects helm-charts repository to be at the root level, so when I changed by repo from

3594XXXXX.dkr.ecr.us-west-2.amazonaws.com/v2/my-project/my-team/helm-charts

to

3594XXXXX.dkr.ecr.us-west-2.amazonaws.com/helm-charts

The push worked fine for me

ubuntu@c2d4a3dd14fd:~/my-bundle/helm-charts/mychart$ helm chart push 359XXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/helm-charts:mychart_0.0.0_test
The push refers to repository [359XXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/helm-charts]
ref:     359XXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/helm-charts:mychart_0.0.0_test
digest:  273040563a534eba7XXXXafb315ed7df70ce384514cd250b5f725ecd5e3
size:    41.3 KiB
name:    mychart
version: 2.0.0
mychart_0.0.0_test: pushed to remote (1 layer, 41.3 KiB total)

Answer 4

Check if you are logged in with helm

Use helm registry login command to login to registry by passing registry address with port(if port is not default)

Login to registry using helm command

$ helm registry login -u myuser registrydomain:5000
Password:
Login succeeded

Logout

$ helm registry logout registrydomain:5000
Logout succeeded

Official Documentation: https://helm.sh/docs/topics/registries/#the-registry-subcommand

Answer 5

If you're getting the following error:

Error: failed to do request: Post "https://account-id.dkr.ecr.us-east-1.amazonaws.com/v2/foo/bar/abc/abc/blobs/uploads/": EOF

See this:

I was having the same problem and figured it out.

The chart's name declared in Chart.yaml file must be the same as the name of the ECR repository. Only then one can execute

helm push foobar-X.Y.Z.tgz oci://.dkr.ecr..amazonaws.com/ and delegate ECR to know in which repository to store the chart.

https://github.com/helm/helm/issues/11017#issuecomment-1244059106

Thank you to manuelnucci