k3s pods are not mounting secrets defined in helm deployment's imagePullSecrets

Question

I am creating a deployment in circlci that deploys my containerized application to a k3s server I have set up. I have set up a secret using the commands found here.

The secret is created using the command:

kubectl create secret docker-registry regkeyname --docker-server=https://index.docker.io/v1/ \
 --docker-username=username  \
 --docker-password=password  \
 --docker-email=my@email.com \
 --namespace=external

My secret is as follows when running kubectl get secret regkeyname --namespace=external --output=yaml:

apiVersion: v1
data:
  .dockerconfigjson: secretbase64stuff
kind: Secret
metadata:
  creationTimestamp: "2020-11-24T13:11:07Z"
  managedFields:
    - apiVersion: v1
      fieldsType: FieldsV1
      fieldsV1:
        f:data:
          .: {}
          f:.dockerconfigjson: {}
        f:type: {}
      manager: kubectl
      operation: Update
      time: "2020-11-24T13:11:07Z"
  name: regkeyname
  namespace: external
  resourceVersion: "16929381"
  selfLink: /api/v1/namespaces/external/secrets/regkeyname
  uid: 51b87508-9cf2-490b-b871-0b5a342ab64c
type: kubernetes.io/dockerconfigjson

I'm using helm to deploy my application and the Deployment looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.labels.app }}
  labels:
    app: {{ .Values.labels.app }}
spec:
  selector:
    matchLabels:
      app: {{ .Values.labels.app }}
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: {{ .Values.labels.app }}
        env: {{ .Values.labels.env }}
    spec:
      imagePullSecrets:
        - name: regkeyname
      containers:
        - name: my-service
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
          imagePullPolicy: {{ .Values.image.imagePullPolicy }}
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
            failureThreshold: 5
            successThreshold: 1

after deploying however, the images fail to pull and it appears that my secret "regkeyname" is not used/mounted in the pods. the result is as follows:

Name:           my-service-856454c6cd-qcp7w
Namespace:      external
Priority:       0
Node:           worker-2/192.168.1.13
Start Time:     Tue, 24 Nov 2020 07:20:08 -0600
Labels: app=my-service
  env=development
  pod-template-hash=856454c6cd
Annotations:    <none>
Status:         Pending
IP:             10.42.2.196
Controlled By:  ReplicaSet/my-service-856454c6cd
Containers:
  auth-service:
    Container ID:
    Image:          my-repo/my-service:latest
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Readiness:      http-get http://:8080/health delay=10s timeout=1s period=10s #success=1 #failure=5
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-l9b4k (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-l9b4k:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-l9b4k
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
  node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From                  Message
  ----     ------     ----                   ----                  -------
  Normal   Scheduled  <unknown>              default-scheduler     Successfully assigned external/auth-service-856454c6cd-qcp7w to worker-2
  Normal   Pulling    32m (x4 over 34m)      kubelet, worker-2  Pulling image "my-repo/my-service:latest"
  Warning  Failed     32m (x4 over 34m)      kubelet, worker-2  Failed to pull image "my-repo/my-service:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/my-repo/my-service:latest": failed to resolve reference "docker.io/my-repo/my-service:latest": failed to do request: Head https://registry-1.docker.io/v2/my-repo/my-service/manifests/latest: dial tcp: lookup registry-1.docker.io: Try again
  Warning  Failed     32m (x4 over 34m)      kubelet, worker-2  Error: ErrImagePull
  Warning  Failed     31m (x6 over 34m)      kubelet, worker-2  Error: ImagePullBackOff
  Normal   BackOff    3m54s (x127 over 34m)  kubelet, worker-2  Back-off pulling image "my-repo/my-service:latest"

I had this working when running locally with kubernetes so I am assuming the issue must have something to do either with k3s or the fact that now the server is remote rather than local. Any insight would be greatly appreciated. Thanks in advance!

score 2 · Answer 1 · answered Nov 24 '20 at 14:40

2

The controller is trying to pull image from the official docker registry:

failed to resolve reference "docker.io/my-repo/my-service:latest"

While creating the imagePullSecret, make sure that you put the correct URL (ie. the URL for your private registry) for performing authentication and pulling image.

$ cat ~/.docker/config.json 
{
    "auths": {
        "https://index.docker.io/v1/": { # <------ change here
            "auth": "..........="
        }
    },
    "HttpHeaders": {
        "User-Agent": "Docker-Client/19.03.5 (linux)"
    }

answered Nov 24 '20 at 14:40

Kamol Hasan

12,218
1
37
46

The repo is private on dockerhub. Does that mean I need to change the docker-server to https://index.docker.io/v1/my-repo? I'm a bit confused at the ~/.docker/config.json. I thought the secret should be used instead of the docker config, is that not how imagePullSecrets work? – Marcus Ruddick Nov 24 '20 at 15:11
Yes, the secret will be used, but the error is saying, the registry don't have an image named `my-servic` in `my-repo` repository. `my-repo` should be replaced by your docker username, if it is an account of dockerhub. – Kamol Hasan Nov 24 '20 at 16:09
This is where my main confusion comes from. I can run docker pull my-repo/my-service:latest locally and it works no problem: it pulls successfully and prints: "Status: Downloaded newer image for my-repo/my-service:latest" and "docker.io/my-repo/my-service:latest" indicating that the repo exists and is found without issue locally. Why is it saying it cant find the repo only from k3s? – Marcus Ruddick Nov 24 '20 at 16:41
@MarcusRuddick It is because you have built the image with that `repo/name`. It finds the image locally, that is why it never pulls the image from the remote server. Try removing(ie. docker rmi ) the image from the local machine, you will find the same error there too. – Kamol Hasan Nov 24 '20 at 17:26

k3s pods are not mounting secrets defined in helm deployment's imagePullSecrets

1 Answers1