4

I am working through a lab that shows how to set-up Kubernetes and the CLI on IBM Cloud.

I have the Kubernetes cluster setup, and the container registry. I am logged in to IBM Cloud and the Container Registry on the CLI. The image has been created and pushed.

I can create a pod using the image with an imperative command using:

kubectl create -f hello-world-create.yaml

where the yaml file looks like:

apiVersion: v1
kind: Pod
metadata:
  name: hello-world
spec:
  containers:
  - name: hello-world
    image: us.icr.io/earlyprogramimages/hello-world:1
    ports:
    - containerPort: 80
  imagePullSecrets:
  - name: icr

but when I try the declarative command for the same image running

kubectl apply -f hello-world-apply.yaml

where the yaml file looks like

apiVersion: apps/v1
kind: Deployment
metadata:
  generation: 1
  labels:
    run: hello-world
  name: hello-world
spec:
  replicas: 3
  selector:
    matchLabels:
      run: hello-world
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        run: hello-world
    spec:
      containers:
      - image: us.icr.io/earlyprogramimages/hello-world:1
        imagePullPolicy: Always
        name: hello-world
        ports:
        - containerPort: 80
          protocol: TCP
      imagePullSecrets:
      - name: icr
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30

I get status of ErrImagePull for each of the pods where the event stack is

Successfully assigned default/hello-world-6fd8bd67dc-79gbz to xx.xx.xx.xx
Pulling image "us.icr.io/earlyprogramimages/hello-world:1

Failed to pull image "us.icr.io/earlyprogramimages/hello-world:1": rpc error: code = Unknown desc = failed to pull and unpack image "us.icr.io/earlyprogramimages/hello-world:1": failed to resolve reference "us.icr.io/earlyprogramimages/hello-world:1": failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized

Error: ErrImagePull

Clearly the command doesn't have read access to image, but I have logged in successfully using

ibmcloud cr login

and can deploy a pod using the imperative create command.

I have been through the documentation, but can't determine which step I have overlooked. What are the extra steps needed which grant the appropriate access for the declarative apply command?

Running

kubectl get secrets -n default | grep "icr-io"

gives

kubectl get secrets -n default | grep "icr-io"
all-icr-io            kubernetes.io/dockerconfigjson        1      167m
default-au-icr-io     kubernetes.io/dockerconfigjson        1      167m
default-de-icr-io     kubernetes.io/dockerconfigjson        1      167m
default-icr-io        kubernetes.io/dockerconfigjson        1      167m
default-jp-icr-io     kubernetes.io/dockerconfigjson        1      167m
default-uk-icr-io     kubernetes.io/dockerconfigjson        1      167m
default-us-icr-io     kubernetes.io/dockerconfigjson        1      167m
chughts
  • 4,210
  • 2
  • 14
  • 27

3 Answers3

6

Here's what I did and worked as expected,

As you can see all-icr-io is the default image pull secret provided in your cluster. Not sure why you were using icr

By default, the IBM Cloud Kubernetes cluster is set up to pull images from only your account’s namespace in IBM Cloud Container Registry by using the secret all-icr-io in the default namespace.

Check the documentation here to copy the existing image pull secret to a non-default namespace

So, my hello-world-create looks like this

apiVersion: v1
kind: Pod
metadata:
  name: hello-world
spec:
  containers:
  - name: hello-world
    image: us.icr.io/mods15/hello-world:1
    ports:
    - containerPort: 80
  imagePullSecrets:
  - name: all-icr-io

and my hello-world-apply.yaml is

apiVersion: apps/v1
kind: Deployment
metadata:
  generation: 1
  labels:
    run: hello-world
  name: hello-world
spec:
  replicas: 3
  selector:
    matchLabels:
      run: hello-world
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        run: hello-world
    spec:
      containers:
      - image: us.icr.io/mods15/hello-world:1
        imagePullPolicy: Always
        name: hello-world
        ports:
        - containerPort: 80
          protocol: TCP
      imagePullSecrets:
      - name: all-icr-io
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30

Here's the outcome once the yaml files are configured successfully enter image description here

Vidyasagar Machupalli
  • 2,737
  • 1
  • 19
  • 29
1

Check out https://cloud.ibm.com/docs/containers?topic=containers-registry#cluster_registry_auth for details on what might be wrong.. Some things to check:

  1. Do you have IAM policies in place that grant you access to the container registry?
  2. Does kubectl get secrets -n default | grep "icr-io" show any pull secrets? If not, follow that doc link above to fix it.
John Pape
  • 106
  • 3
  • I have updated the question with the output from the command. I am checking the documentation link you provided for clues. – chughts Aug 18 '20 at 16:38
  • Found something related to when clusters were created. My Kubernetes cluster is new, but my registry namespace is quite old. I think I will drop and recreate the namespace. – chughts Aug 18 '20 at 17:02
  • Created a brand new cluster and brand new registry namespace - same issue. – chughts Aug 18 '20 at 21:09
0

I had this issue as well but on Azure. I tried everything but nothing helped or was already configured that way. What worked for me was to downgrade my Linux version. After a lot of digging I found that Azure Kubernetes Service runs on 16 and 18 so I chose one of those builds and it worked.

Kasey Chakos
  • 75
  • 1
  • 7