Microk8s cannot pull from private registry

Question

I'm running Microk8s on an EC2 instance. I fail to pull containers from our private registry. When trying to run such a container kubectl describe pod shows:

Failed to pull image "docker.xxx.com/import:v1": rpc error: code = Unknown desc = failed to resolve image "docker.xxx.com/import:v1": no available registry endpoint: failed to fetch anonymous token: unexpected status: 401 Unauthorized

I can docker login and docker pull from that machine. The yaml I used to deploy the container is working fine on another (non containerd) cluster. It refers to a pull secret, which is identical to the one used in the other cluster and working fine there.

I added the following entry to the containerd-template.toml of Microk8s:

   [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        ...
        [plugins.cri.registry.mirrors."docker.xxx.com"]
          endpoint = ["https://docker.xxx.com"]

I have no idea what else I might be missing.

Did you enable registry? `microk8s.enable registry` Did you edit deamon.json? `sudo vim /etc/docker/daemon.json` with entry like `{ "insecure-registries" : ["yourRepository"] }' — PjoterS, Jun 21 '19 at 09:21

score 2 · Answer 1 · answered Jun 27 '19 at 09:40

If you are getting an error 401 probably something is wrong with the authentication. E.g. you are missing credentials to your private registry.

To make sure that microk8s would use proper credentials, in addition of mirrors sections within the configuration you have to specify auths section where you would put your docker registry credentials.

[plugins.cri.registry.auths]
  [plugins.cri.registry.auths."https://gcr.io"]
    username = ""
    password = ""
    auth = ""
    identitytoken = ""

Attributes within that section are compatible with configuration which you can find in your .docker/config.json.

Notice that this is section on the same level as mirrors it should not be part of mirrors entry but added as new section. Another important part is to make sure that the auth hosts match yours registry host (e.g. https vs http).

For more details check reference: https://github.com/containerd/cri/blob/master/docs/registry.md

p.s. Keep in mind that containerd is supported from microk8s[1] v1.14 if you use older version you should check other options like official kubernates documentation[2]

[1] https://microk8s.io/docs/working

[2] https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

score 1 · Answer 2 · answered Nov 19 '19 at 23:33

The simplest solution would be to:

1) create a secret in the current namespace:

k create secret docker-registry my-private-registry \
  --docker-server=my-private-registry.io \
  --docker-username="my-username" \
  --docker-password="********" \
  --docker-email=my.email@helloworld.io

2) patch the default service account:

k patch serviceaccount default -p '{"imagePullSecrets": [{"name": "my-private-registry"}]}'

After this you can reference images from your private docker registry in your pod/deployment/replicaset definitions. Perform the same sequence for every namespace you'll be using.

HTH.

Microk8s cannot pull from private registry

2 Answers2