0

I'm running an on-premise K8s cluster. The problem is Kubernetes doesn't see the locally built container images, despite the images being visible in the k8s.io namespace - tested both with nerdctl --namespace=k8s.io images and crictl images commands. crictl guide

Cluster information:

Kubernetes version: 1.26.1

on-premise cluster

Host OS: AlmaLinux 9.1

CNI and version: Calico v3.24.5

Container Runtime: Containerd v1.6.16

[root@alma-kube ~]# kubectl get nodes
NAME         STATUS   ROLES           AGE   VERSION
alma-kube    Ready    control-plane   62d   v1.26.1
alma-kube2   Ready    worker          61d   v1.26.0

crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
...
docker.io/library/mytestimage             latest              f240b96f676e1       2.81MB
docker.io/library/nginx-test              1                   5e34a0811eefb       56.9MB
docker.io/library/nginx-test              latest              5e34a0811eefb       56.9MB
docker.io/nick/nginx                      latest              5e34a0811eefb       56.9MB
docker.io/registry-nick/nginx-test        latest              5e34a0811eefb       56.9MB

nerdctl --namespace=k8s.io images
REPOSITORY                                 TAG        IMAGE ID        CREATED         PLATFORM       SIZE         BLOB SIZE
...
mytestimage                                latest     dfb4f2a8f7c9    5 hours ago     linux/amd64    5.6 MiB      2.7 MiB
nginx-test                                 1          7f7e7eb6b3c2    2 days ago      linux/amd64    146.7 MiB    54.3 MiB
registry                                   latest     3f71055ad7c4    2 days ago      linux/amd64    23.7 MiB     8.8 MiB
registry-nick/nginx-test                   latest     7f7e7eb6b3c2    3 days ago      linux/amd64    146.7 MiB    54.3 MiB

The Pod YAML file:

Note: The commented out "#image:" lines are things I tried and all of them failed with the same [error](### The pod fails to start with ErrImageNeverPull error:). I tried accessing the image directly - this is my primary goal. Because that didn't work, I also tried by running a private registry.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-test-pod
spec:
  containers:
  - name: nginx-test-crp
    image: mytestimage
    #image: docker.io/library/nginx-test:1
    #image: registry-nick/nginx-test
    #image: localhost:5000/nginx-test
    imagePullPolicy: Never

I create the pod using the command kubectl apply -f pod-test.yaml

The pod fails to start with ErrImageNeverPull error:

kubectl get pod -o wide --all-namespaces
default            nginx-test-pod                             0/1     ErrImageNeverPull   0               124m   192.168.48.11    alma-kube2   <none>  

kubectl logs nginx-test-pod
Error from server (BadRequest): container "nginx-test-crp" in pod "nginx-test-pod" is waiting to start: ErrImageNeverPull

kubectl describe pod nginx-test-pod
Events:
  Type     Reason             Age                   From     Message
  ----     ------             ----                  ----     -------
  Warning  ErrImageNeverPull  58s (x630 over 136m)  kubelet  Container image "local-nick/bokluk" is not present with pull policy of Never

Obviously K8s can't see the locally built image. How can I fix this problem?

Nikolas Naydenov
  • 1
  • 1
  • What have you done to transfer the image onto the worker node? (Obviously it cannot access your local registry server while you have it configured not to pull from the registry.) – benjimin Feb 25 '23 at 21:31
  • Yes, thanks, I forgot that. Is there some automated way to share image cache berween cluster nodes? – Nikolas Naydenov Feb 25 '23 at 22:12
  • Yes, problem was fixed by copying the image to the worker node: alma-kube ~]# nerdctl --namespace=k8s.io image save nginx-test:1 -o nginx-test-1.tar alma-kube2 ~]# nerdctl --namespace=k8s.io load -i nginx-test-1.tar I'm looking for a way to share the image acache between all nodes by default. If no such functionality is supported, I guess the private registry is the only way. – Nikolas Naydenov Feb 25 '23 at 23:40

1 Answers1

0

You need to transfer the image onto the worker node.

A local registry will not work while you have the pod configured with imagePullPolicy: Never.

If you want to keep your container images on-premises then either:

  • set up a local image registry server and relax the image pull policy, or
  • pre-load the image onto your worker nodes (perhaps using their launch scripts).
benjimin
  • 4,043
  • 29
  • 48