skaffold and microk8s -- container server is waiting to start: can't be pulled

Question

I'm hitting this error on a unbuntu 20.04 installation, with microk8s and skaffold.

container server is waiting to start: aliwatters/dkc-multi-server:7b9b7d87ab81483cdb9b1227780d68dec815fe271aea2f9fe3a0e65090d90909 can't be pulled

I'm logged into docker.

Full error is

exiting dev mode because first deploy failed: 1/1 deployment(s) failed
ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ skaffold dev
Listing files to watch...
 - aliwatters/dkc-multi-server
Generating tags...
 - aliwatters/dkc-multi-server -> aliwatters/dkc-multi-server:90d1283
Checking cache...
 - aliwatters/dkc-multi-server: Found Locally
Tags used in deployment:
 - aliwatters/dkc-multi-server -> aliwatters/dkc-multi-server:7b9b7d87ab81483cdb9b1227780d68dec815fe271aea2f9fe3a0e65090d90909
Starting deploy...
 - service/server-cluster-ip-service created
 - deployment.apps/server-deployment created
Waiting for deployments to stabilize...
 - deployment/server-deployment: creating container server
    - pod/server-deployment-5f496cf9c7-8dfp8: creating container server
    - pod/server-deployment-5f496cf9c7-hzc2l: creating container server
    - pod/server-deployment-5f496cf9c7-2mn6x: container server is waiting to start: aliwatters/dkc-multi-server:7b9b7d87ab81483cdb9b1227780d68dec815fe271aea2f9fe3a0e65090d90909 can't be pulled
 - deployment/server-deployment failed. Error: creating container server.
Cleaning up...
 - service "server-cluster-ip-service" deleted
 - deployment.apps "server-deployment" deleted
exiting dev mode because first deploy failed: 1/1 deployment(s) failed

All the relevant files;

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ cat skaffold.yaml 
apiVersion: skaffold/v2beta11
kind: Config
build:
  artifacts:
    - image: aliwatters/dkc-multi-server
      context: server
      docker:
        dockerfile: Dockerfile.dev
      sync:
        manual:
        - src: '**/*.js'
          dest: .
  local:
    push: false
deploy:
  kubectl:
    manifests:
      - k8s/server-cluster-ip-service.yaml
      - k8s/server-deployment.yaml

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ cat k8s/server-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: server-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      component: server
  template:
    metadata:
      labels:
        component: server
    spec:
      containers:
        - name: server
          image: aliwatters/dkc-multi-server
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5000
          env:
            - name: REDIS_HOST
              value: redis-cluster-ip-service
            - name: REDIS_PORT
              value: '6379'
            - name: PGUSER
              value: postgres
            - name: PGHOST
              value: postgres-cluster-ip-service
            - name: PGPORT
              value: '5432'
            - name: PGDATABASE
              value: postgres
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: pgpasswd
                  key: PGPASSWORD

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ cat k8s/server-cluster-ip-service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: server-cluster-ip-service
spec:
  type: ClusterIP
  selector:
    component: server
  ports:
    - port: 5000
      targetPort: 5000

And the microk8s config

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ more ~/.kube/config 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBVENDQWVtZ0F3SUJBZ0lKQUx
aR3B0NDZ... snip ..EhqYThFZjBsMzVFdmxFa1pJeGpwOHh5NEVLUHMKTFpCbDB3TT0KLS0tLS1FTkQgQ0VSV
ElGSUNBVEUtLS0tLQo=
    server: https://127.0.0.1:16443
  name: microk8s-cluster
contexts:
- context:
    cluster: microk8s-cluster
    user: admin
  name: microk8s
current-context: microk8s
kind: Config
preferences: {}
users:
- name: admin
  user:
    token: VDlqa3BnWlR4Q3UzaHhpQnFvamlkbDdBRk9aMzdBcDkvMFhZT21ad1VqND0K

Versions

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ snap list
Name                  Version                     Rev    Tracking         Publisher    Notes
kubectl               1.20.2                      1811   latest/stable    canonical✓   classic
microk8s              v1.20.1                     1910   1.20/stable      canonical✓   classic

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.1-34+e7db93d188d0d1", GitCommit:"e7db93d188d0d12f2fe5336d1b85cdb94cb909d3", GitTreeState:"clean", BuildDate:"2021-01-11T23:48:42Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.1-34+e7db93d188d0d1", GitCommit:"e7db93d188d0d12f2fe5336d1b85cdb94cb909d3", GitTreeState:"clean", BuildDate:"2021-01-11T23:50:46Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"linux/amd64"}

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ skaffold version
v1.18.0

Any ideas?

Try to [Create a Secret based on existing Docker credentials](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials) and [Create a Pod that uses your Secret](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-pod-that-uses-your-secret) — Matt, Feb 01 '21 at 09:28
Thanks Matt, I ran through those, skeptically -- as this is a not a private image, no change. https://hub.docker.com/repository/registry-1.docker.io/aliwatters/dkc-multi-server/tags?page=1&ordering=last_updated -- notice that the tag asked for by skaffold isn't present. — Ali W, Feb 01 '21 at 15:50
https://github.com/GoogleContainerTools/skaffold/issues/5327 -- turns out that at this moment in time microk8s is not fully supported by skaffold. (seeing if I can fix that) — Ali W, Feb 03 '21 at 15:37

score 0 · Answer 1 · answered May 25 '22 at 08:18

This happens because MicroK8s doesn't see the images which were built by your local Docker daemon. The final solution to this problem is being discussed in this GitHub issue.

For the moment, you can solve the problem by importing your images into MicroK8s:

docker save myimage > myimage.tar;
microk8s ctr image import myimage.tar;

If you prefer a global solution, you can start MicroK8s built-in registry with microk8s enable registry. This will expose as a NodePort service on port 32000 of the localhost.

Then run your Skaffold commands with default-repo functionality by using --default-repo or just -d:

skaffold run -d localhost:32000

skaffold and microk8s -- container server is waiting to start: can't be pulled

1 Answers1