Error imagepullbackoff when scaling working deployment Kubernetes

Question

I have a working deployment of a docker image on Kubernetes. However, when I want to scale it I recieve and error that it cannot find my image (even though I'm scaling something that's already working from an image?)

Here is the command I'm using to scale the deployment.

./kubectl scale deployments/mautic --replicas=2

Here is the logs when I run Kubectl describe

Name:           mautic-3389378641-jgm9b
Namespace:      default
Node:           minikube/192.168.99.101
Start Time:     Fri, 01 Sep 2017 14:34:08 +0100
Labels:         app=mautic
                pod-template-hash=3389378641
                tier=frontend
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"mau
tic-3389378641","uid":"52a87ff6-8f06-11e7-8fbc-080027cd66fa",...
Status:         Pending
IP:             172.17.0.8
Created By:     ReplicaSet/mautic-3389378641
Controlled By:  ReplicaSet/mautic-3389378641
Containers:
  mautic:
    Container ID:
    Image:              mautic/mautic:latest
    Image ID:
    Port:               80/TCP
    State:              Waiting
      Reason:           ErrImagePull
    Ready:              False
    Restart Count:      0
    Environment:
      MAUTIC_DB_HOST:           mautic-mysql
      MAUTIC_DB_PASSWORD:       <set to the key 'password.txt' in secret 'mysql-pass'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-9drh0 (ro)
      /var/www/html from mautic-local-storage (rw)
Conditions:
  Type          Status
  Initialized   True
  Ready         False
  PodScheduled  True
Volumes:
  mautic-local-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mautic-lv-claim
    ReadOnly:   false
  default-token-9drh0:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-9drh0
    Optional:   false
QoS Class:      BestEffort
Node-Selectors: <none>
Tolerations:    <none>
Events:
  FirstSeen     LastSeen        Count   From                    SubObjectPath           Type            Reason                  Message
  ---------     --------        -----   ----                    -------------           --------        ------                  -------
  47m           47m             1       default-scheduler                               Normal          Scheduled               Successfully assigned mauti
c-3389378641-jgm9b to minikube
  47m           47m             1       kubelet, minikube                               Normal          SuccessfulMountVolume   MountVolume.SetUp succeeded
 for volume "pvc-5b7c14a3-8f03-11e7-8fbc-080027cd66fa"
  47m           47m             1       kubelet, minikube                               Normal          SuccessfulMountVolume   MountVolume.SetUp succeeded
 for volume "default-token-9drh0"
  47m           7m              12      kubelet, minikube       spec.containers{mautic} Normal          Pulling                 pulling image "mautic/mauti
c:latest"
  47m           6m              12      kubelet, minikube       spec.containers{mautic} Warning         Failed                  Failed to pull image "mauti
c/mautic:latest": rpc error: code = 2 desc = Network timed out while trying to connect to https://index.docker.io/v1/repositories/mautic/mautic/images. You
 may want to check your internet connection or if you are behind a proxy.
  47m           6m              170     kubelet, minikube                               Warning         FailedSync              Error syncing pod
  47m           6m              158     kubelet, minikube       spec.containers{mautic} Normal          BackOff                 Back-off pulling image "mau
tic/mautic:latest"

But the Mautic image referenced is right here and is already in use with the deployment I want to scale.

REPOSITORY                                             TAG                 IMAGE ID            CREATED             SIZE
testimage                                              v0                  0e0d4b13c0c2        10 days ago         611MB
mautic/mautic                                          latest              730d2796f904        2 weeks ago         611MB
mysql                                                  5.6                 cdfa8cc50c33        5 weeks ago         298MB
mysql                                                  latest              c73c7527c03a        5 weeks ago         412MB
gcr.io/google_containers/k8s-dns-sidecar-amd64         1.14.4              38bac66034a6        2 months ago        41.8MB
gcr.io/google_containers/k8s-dns-kube-dns-amd64        1.14.4              a8e00546bcf3        2 months ago        49.4MB
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64   1.14.4              f7f45b9cb733        2 months ago        41.4MB
gcr.io/google-containers/kube-addon-manager            v6.4-beta.2         0a951668696f        2 months ago        79.2MB
gcr.io/google_containers/kubernetes-dashboard-amd64    v1.6.1              71dfe833ce74        3 months ago        134MB
autoize/mautic                                         latest              6c99d7ce1a07        4 months ago        665MB
gcr.io/google_containers/pause-amd64                   3.0                 99e59f495ffa        16 months ago       747kB

Pods

Does anyone have any idea why this isn't working?

Do you mean a node? I'm new to Kubernetes. I'm running this using minikube from a docker-machine virtualbox vm if that helps. — John Deegs, Sep 01 '17 at 14:33
Yeah I meant node. If you had more than one node (not your case) it could have been possible that one node had the image available, and the other didn't. Anyway, ensure you have connectivity and try again, looks like a temporary problem to me. `Network timed out while trying to connect to...` — whites11, Sep 01 '17 at 14:35
I have connectivity, my minikube is running and I can access my dashboard and the original instance that's running (the one I'm looking to scale). Still getting the same error. Any more ideas? Thanks for replying :). — John Deegs, Sep 01 '17 at 14:39
No more ideas, try giving it a few minutes or even try restarting minikube if you can. — whites11, Sep 01 '17 at 14:41
it's really hard to argue with that kind of an error message.. can you confirm that with replica=1 everything is fine? and by find I mean do the describe again and see it's okay, also do `kubectl get pods` and make sure the pod is alive and running, let me know. — Naim Salameh, Sep 01 '17 at 20:02

score 0 · Answer 1 · answered Sep 03 '17 at 18:29

Your host has lost connectivity to Docker Hub - try running "docker pull mautic/mautic:latest" and see if that works. This may be a network issue on your host, a problem with an intermediate proxy between your host and Docker Hub or (less likely) a temporary outage at Docker Hub or something else.

Since you're using the latest tag it is likely that your Deployment is using imagePullPolicy=Always. (docs: Defaults to Always if :latest tag is specified, or IfNotPresent otherwise.) I'd suggest explicitly setting imagePullPolicy=IfNotPresent in your Deployment spec so that the local image that already exists is used when starting a new container.

Error imagepullbackoff when scaling working deployment Kubernetes

1 Answers1