3

I have a simple setup with docker-desktop for mac, with the Kubernetes cluster that comes with it.

Every time I run skaffold dev, with imagePullPolicy: Never, an image gets created, but I still get an error when deploying the dirty image to the Kubernetes cluster:

ErrImageNeverPull: Container image "user/test:bf13343a4b431df2b7df84f58ede9229bd3d868a4b10147fff04eae8a7adc0cd" is not present with pull policy of Never

Can anyone help me debug this? I have no idea how to investigate this further. But skaffold is not deploying my dirty image to the Kubernetes cluster. And everyone is suggesting adding imagePullPolicy: Never to my deployment.yaml but that doesn't fix anything

The skaffold.yml content:

apiVersion: skaffold/v2beta5
kind: Config
metadata:
  name: test
build:
  artifacts:
  - image: user/test
deploy:
  kubectl:
    manifests:
    - kube/postgres/postgres-configmap.dev.yaml
    - kube/redis/redis-configmap.secret.yaml
    - kube/web/web-configmap.dev.yaml
    - kube/elasticsearch/elasticsearch-storage.dev.yaml
    - kube/postgres/postgres-storage.dev.yaml
    - kube/cable/cable-service.yaml
    - kube/dashboard.yaml
    - kube/cable/cable-deployment.yaml
    - kube/nginx/load-balancer.dev.yaml
    - kube/redis/redis-storage.yaml
    - kube/sidekiq/sidekiq-deployment.yaml
    - kube/web/web-deployment.yaml
    - kube/web/web-service.dev.yaml
    - kube/web/web-migration.yaml
    - kube/web/web-seed.yaml
Cosmin Atanasiu
  • 2,532
  • 3
  • 21
  • 26
  • please share the whole logs of `skaffold dev` : I want to check context and others.. also, if you could share the content of skaffold yaml file , it will be better – Abdennour TOUMI Jul 13 '20 at 17:50
  • And the logs from running `skaffold dev -v debug`. And what happens if you run `docker images user/test:bf13343a4b431df2b7df84f58ede9229bd3d868a4b10147fff04eae8a7adc0cd`? – Brian de Alwis Jul 13 '20 at 20:49
  • I added the skaffold.yaml, but as far as the logs go everything looks good, and then I get `ErrImageNeverPull` errors repeatedly until `skaffold dev` rolls back entirely. – Cosmin Atanasiu Jul 14 '20 at 09:40

1 Answers1

2

The only thing I can think of is that your skaffold.yaml doesn't match the image name:

apiVersion: skaffold/v2beta5
kind: Config
build:
  artifacts:
  - image: user/test  does this match?
deploy:
  kubectl:
    manifests:
      - deployment.yaml

Otherwise, something might be corrupted with your docker + K8s instance. You can try:

  • Enable/Disable K8s in Docker
  • Use KinD
Rico
  • 58,485
  • 12
  • 111
  • 141
  • Actually yes, the `build.artifacts.image` does match the image name. And the image does get created locally, so when I do `docker image ls` I see the newly created dirty image. The error is given by docker-desktop's kubernetes node on mac (not minikube): `pod/web-7f685f866c-ncwt8: ErrImageNeverPull: Container image "user/test:bf13343a4b431df2b7df84f58ede9229bd3d868a4b10147fff04eae8a7adc0cd" is not present with pull policy of Never` – Cosmin Atanasiu Jul 14 '20 at 09:14
  • I keep thinking that for some reason the kubernetes node can't access my local images. And if I try `imagePullPolicy:Always` or `IfNotPresent`, then I get this error, since I'm using a private image repo: `Failed: Failed to pull image "user/test:59f07723660cc1b012440144cf9f75512da07d7e266aada9af8aea97f12089ff": rpc error: code = Unknown desc = Error response from daemon: manifest for user/test:59f07723660cc1b012440144cf9f75512da07d7e266aada9af8aea97f12089ff not found: manifest unknown: manifest unknown` – Cosmin Atanasiu Jul 14 '20 at 09:18
  • 1
    There might be something corrupted about the K8s cluster in your docker instances. You can try a couple of things: 1) disable and reenable K8s in docker. 2) Try to use KinD. – Rico Jul 14 '20 at 14:42
  • This may well be it. Unfortunately I'm not experienced enough to debug this, so rather than uninstalling and reinstalling, I've disabled kubernetes in `docker-desktop` and installed `minikube`, then skaffold ran well for a good half hour, then threw this error `Failed to validate 'docker' driver`.. Just need to fix this and I think I'm good to go – Cosmin Atanasiu Jul 15 '20 at 09:51
  • Yup, just also needed to uncheck `Deploy Docker Stacks to Kubernetes by default` in docker-desktop preferences and now the system is running – Cosmin Atanasiu Jul 15 '20 at 09:57