Kubernetes pull from local registry not working, how to fix that?

Question

I followed this link and created a secret as below.

kubectl create secret docker-registry regsec --docker-server=192.168.56.106:5000 --docker-username=osboxes --docker-password=osboxes.org --insecure-skip-tls-verify=true

And the deployment as below.

kubectl create deploy nginx1 --image 192.168.56.106:5000/todoapp:1.0

And edited it using

kubectl edit deploy nginx1

And added imagePullSecrets to it

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
  creationTimestamp: "2021-07-21T10:23:23Z"
  generation: 2
  labels:
    app: nginx1
  name: nginx1
  namespace: default
  resourceVersion: "6872"
  uid: 0b6917f0-10ac-4206-82a8-c49ae8ffa2b3
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx1
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx1
    spec:
      containers:
      - image: 192.168.56.106:5000/todoapp:1.0
        imagePullPolicy: IfNotPresent
        name: todoapp
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: regsec
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  conditions:
  - lastTransitionTime: "2021-07-21T10:23:23Z"
    lastUpdateTime: "2021-07-21T10:23:23Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2021-07-21T10:23:23Z"
    lastUpdateTime: "2021-07-21T10:28:36Z"
    message: ReplicaSet "nginx1-75df7fd466" is progressing.
    reason: ReplicaSetUpdated
    status: "True"
    type: Progressing
  observedGeneration: 2
  replicas: 2
  unavailableReplicas: 2
  updatedReplicas: 1

But still getting below error.

osboxes@osboxes:~/Desktop$ kubectl get pods
    NAME                      READY   STATUS             RESTARTS   AGE
    nginx-6799fc88d8-vvzjf    1/1     Running            0          19m
    nginx1-65d848d94f-dd4ck   0/1     ImagePullBackOff   0          12m
    nginx1-75df7fd466-kn5mf   0/1     ImagePullBackOff   0          6m50s

osboxes@osboxes:~/Desktop$ kubectl describe pod nginx1-75df7fd466-kn5mf
Name:         nginx1-75df7fd466-kn5mf
Namespace:    default
Priority:     0
Node:         samples-control-plane/172.19.0.3
Start Time:   Wed, 21 Jul 2021 06:28:36 -0400
Labels:       app=nginx1
              pod-template-hash=75df7fd466
Annotations:  <none>
Status:       Pending
IP:           10.244.0.8
IPs:
  IP:           10.244.0.8
Controlled By:  ReplicaSet/nginx1-75df7fd466
Containers:
  todoapp:
    Container ID:
    Image:          192.168.56.106:5000/todoapp:1.0
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-88clq (ro)
Conditions:

  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-88clq:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  73s                default-scheduler  Successfully assigned default/nginx1-75df7fd466-kn5mf to samples-control-plane
  Normal   Pulling    33s (x3 over 73s)  kubelet            Pulling image "192.168.56.106:5000/todoapp:1.0"
  Warning  Failed     33s (x3 over 73s)  kubelet            Failed to pull image "192.168.56.106:5000/todoapp:1.0": rpc error: code = Unknown desc = failed to pull and unpack image "192.168.56.106:5000/todoapp:1.0": failed to resolve reference "192.168.56.106:5000/todoapp:1.0": failed to do request: Head "https://192.168.56.106:5000/v2/todoapp/manifests/1.0": http: server gave HTTP response to HTTPS client
  Warning  Failed     33s (x3 over 73s)  kubelet            Error: ErrImagePull
  Normal   BackOff    6s (x4 over 72s)   kubelet            Back-off pulling image "192.168.56.106:5000/todoapp:1.0"
  Warning  Failed     6s (x4 over 72s)   kubelet            Error: ImagePullBackOff

kubectl version

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-13T02:40:46Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-21T23:01:33Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

Adding docker insecure registry is not useful as the container runtime is not docker now.It is containerd.

/etc/default/docker.json

And for reference, I have setup the local container on a virtualbox VM and connecting to kubernetes on the same network using Host-only network ip address.

The setup of local registry is created using the steps mentioned in link.

And the kubernetes was setup using kind.

Answer 1

cri-o

If you are using cri-o as Container Runtime Provider, docker settings are of no use. You would require to configure cri-o instead: https://github.com/cri-o/cri-o#configuration

Assuming you have cri-o installed on the host node, the official documentation recommends to have a $HOME/.config/containers/registries.conf file, or a global, /etc/containers/registries.conf file to configure registries in a runtime engine agnostic way.

The file structure spec is also documented: https://github.com/containers/image/blob/main/docs/containers-registries.conf.5.md

Example:

[registries.search]
registries = ['registry1.com', 'registry2.com']

[registries.insecure]
registries = ['registry3.com']

[registries.block]
registries = ['registry.untrusted.com', 'registry.unsafe.com']

containerd

Containerd does not seem to acknowledge the /etc/containers/registries.conf settings.

As per the documentation, The main config file /etc/containerd/config.toml can specify a registry config path as:

[plugins."io.containerd.grpc.v1.cri".registry]
   config_path = "/etc/containerd/certs.d"

And inside the path specified above (/etc/containerd/certs.d) create a directory for host (docker.io) and in it, create a hosts.toml file. That is /etc/containerd/certs.d/docker.io/hosts.toml:

server = "https://registry-1.docker.io"

[host."http://my-custom-registry:5000"]
  capabilities = ["pull", "resolve", "push"]
  skip_verify = true
  plain-http = true

For my-custom-registry:5000 section you can also provide credentials as well as certificates.

And restart containerd daemon/service on the host.

Configuration file spec is here: https://github.com/containerd/containerd/blob/main/docs/hosts.md