ErrImagePull: "no basic auth credentials" when kubernetes pulls from ECR

Question

I am getting a no basic auth credentials ErrImagePull error when pulling an image hosted on a private ECR registry. I have to my knowledge checked off all the necessary requirements for successful authorization, so I'll list them below in hopes that someone might suggest something I missed.

• aws ecr get-login-password --region eu-west-2 | docker login --username AWS --password-stdin redacted.dkr.ecr.eu-west-2.amazonaws.com is successful

• I can successfully run the image through a docker pull and docker run. Meaning my account has privileges to pull the image

• I have created a separate namespace called client-ns

• I have created a regcred with this command:

    kubectl create secret docker-registry regcred \
      --docker-server=redacted.dkr.ecr.eu-west-2.amazonaws.com \
      --docker-username=AWS \
      --docker-password=$(aws ecr get-login-password) \
      --namespace=client-ns

• In my manifest.yml I have added the variables:

    imagePullSecrets:
    - name: regcred

• I have inspected the secrets and double checked that they match using the commands below

    aws ecr get-login-password

    kubectl get secret regcred --namespace=client-ns --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode

Despite all these steps I still get the error, any ideas what could be causing this? Below are the events of the pod

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  45s                default-scheduler  Successfully assigned client-ns/client-deployment-8567f459b8-9j25j to minikube
  Normal   BackOff    17s (x3 over 43s)  kubelet            Back-off pulling image "redacted.dkr.ecr.eu-west-2.amazonaws.com/client:v1.0.1"
  Warning  Failed     17s (x3 over 43s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    5s (x3 over 44s)   kubelet            Pulling image "redacted.dkr.ecr.eu-west-2.amazonaws.com/client:v1.0.1"
  Warning  Failed     5s (x3 over 44s)   kubelet            Failed to pull image "redacted.dkr.ecr.eu-west-2.amazonaws.com/client:v1.0.1": rpc error: code = Unknown desc = Error response from daemon: Head "https://redacted.dkr.ecr.eu-west-2.amazonaws.com/v2/client/manifests/v1.0.1": no basic auth credentials
  Warning  Failed     5s (x3 over 44s)   kubelet            Error: ErrImagePull

Edit: Below is my deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: client-deployment 
  namespace: client-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: client-app
  template:
    metadata:
      labels:
        app: client-app
    spec:
      containers:
      - name: client-app 
        image: redacted.dkr.ecr.eu-west-2.amazonaws.com/client:v1.0.1
        imagePullPolicy: Always
        ports:
          - containerPort: 50054
            name: eds-app
        securityContext:
          runAsUser: 20000
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
        resources:
          requests:
            memory: "64Mi"
            cpu: "100m"
            ephemeral-storage: "5Mi"
          limits:
            memory: "128Mi"
            cpu: "200m"
            ephemeral-storage: "10Mi"
        volumeMounts:
        - name: env-file
          mountPath: /client/build/default.env
          subPath: default.env
        - name: csv-file
          mountPath: /client/csv/customers.csv
          subPath: customers.csv
      volumes:
      - name: env-file
        hostPath:
          path: /home/nocnoc/Documents/redacted/github/client/build/default.env
      - name: csv-file
        hostPath:
          path: /home/nocnoc/Documents/redacted/github/client/csv/customers.csv
      imagePullSecrets:
      - name: regcred

Answer 1

I had exactly the same problem on my bare-metal k8s 1.27 cluster (1 control plane, 8 workers, containerd runtime, cilium CNI) the problem was a RBAC control on secrets, I got the error Unknown desc = Error response from daemon: Head "……": no basic auth credentials
The deployement was:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: netshell
  namespace: sandbox
  labels:
    app: echo-on-port-8000
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo-on-port-8000
  strategy: {}
  template:
    metadata:
      labels:
        app: echo-on-port-8000
    spec:
      containers:
        - name: netshell
          image: aregistry.example.org/net/net-tools:latest
          imagePullPolicy: Always
          command: ["/bin/sh"]
          args: ["-c", "cd ~/ && touch file.txt && mknod -m 777 fifo p && cat fifo | netcat -k -l 8000 > fifo && sleep infinity"]
          ports:
            - containerPort: 8000
          resources:
            limits:
              cpu: 250m
              memory: "536870912"
      restartPolicy: Always
      imagePullSecrets:
        - name: local-registry-credential
status: {}

I solved it by adding the get,watch,list capabilities on secrets.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: sandbox
  name: secret-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["secrets"]  #grants reading namespace pods and secrets
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: secret-reader
  namespace: sandbox
subjects:
- kind: ServiceAccount
  name: default # "name" is case sensitive
  namespace: sandbox
roleRef:
  kind: Role #this must be Role or ClusterRole
  name: secret-reader 
  apiGroup: rbac.authorization.k8s.io
---