Minikube Failed to pull image http: server gave HTTP response to HTTPS client

Question

I've a minikube running on OSX with hyperkit with insecure registry enabled

$ minikube start
  minikube 1.12.1 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.12.1
  To disable this notice, run: 'minikube config set WantUpdateNotification false'

  minikube v1.9.2 on Darwin 10.15.6
✨  Using the hyperkit driver based on existing profile
  Starting control plane node m01 in cluster minikube
  Restarting existing hyperkit VM for "minikube" ...
  Preparing Kubernetes v1.18.0 on Docker 19.03.8 ...
  Enabling addons: dashboard, default-storageclass, registry, storage-provisioner
  Done! kubectl is now configured to use "minikube"

I can inspect the registry on the minikube IP address

$ curl $(minikube ip):5000/v2/_catalog
{"repositories":[]}
$ curl https://$(minikube ip):5000/v2/_catalog
curl: (35) error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number

I;ve built an image locally and

$ kubectl create deployment spark-k8s --image=192.168.64.9:5000/spark-k8s:latest

If I check the status of the pod I see that it was not able to pull the image, see below

k describe pod spark-k8s-799578f68d-2v54k
Name:         spark-k8s-799578f68d-2v54k
Namespace:    default
Priority:     0
Node:         minikube/192.168.64.9
Start Time:   Thu, 23 Jul 2020 20:30:18 -0700
Labels:       app=spark-k8s
              pod-template-hash=799578f68d
Annotations:  <none>
Status:       Pending
IP:           172.17.0.11
IPs:
  IP:           172.17.0.11
Controlled By:  ReplicaSet/spark-k8s-799578f68d
Containers:
  spark-k8s:
    Container ID:   
    Image:          192.168.64.9:5000/spark-k8s:latest
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-f9p8t (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-f9p8t:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-f9p8t
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  <unknown>              default-scheduler  Successfully assigned default/spark-k8s-799578f68d-2v54k to minikube
  Normal   Pulling    5m10s (x4 over 6m29s)  kubelet, minikube  Pulling image "192.168.64.9:5000/spark-k8s:latest"
  Warning  Failed     5m10s (x4 over 6m29s)  kubelet, minikube  Failed to pull image "192.168.64.9:5000/spark-k8s:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://192.168.64.9:5000/v2/: http: server gave HTTP response to HTTPS client
  Warning  Failed     5m10s (x4 over 6m29s)  kubelet, minikube  Error: ErrImagePull
  Warning  Failed     4m41s (x6 over 6m29s)  kubelet, minikube  Error: ImagePullBackOff
  Normal   BackOff    86s (x20 over 6m29s)   kubelet, minikube  Back-off pulling image "192.168.64.9:5000/spark-k8s:latest"

How can I get kuernetes to use HTTP to pull the image from the insecure local registry?

I regrettably just noticed this, but is there a reason you're running such an outdated version? *minikube 1.12.1 is available!* — mdaniel, Jul 25 '20 at 03:12
I just didn't updated minikube for a while, will upgrade to latest — bachr, Jul 25 '20 at 16:41
@mdaniel it is actually working, I deleted the current minikube VM, upgrade it and did the exact same steps above and my deployment using the local registry just worked!! — bachr, Jul 26 '20 at 03:54

score 2 · Answer 1 · edited Jul 27 '20 at 14:45

2

That's not something kubernetes, itself, controls (AFAIK) -- rather if you're using docker you'll need to enable --insecure-registries which one can do via an argument to the docker daemon, or (more conveniently) via /etc/docker/daemon.json (as shown in that documentation link)

I've never had to do that for containerd, but it likely has a similar concept

As mentioned in comments version of minikube was the issue.

I regrettably just noticed this, but is there a reason you're running such an outdated version? minikube 1.12.1 is available!

And as @bachr confirmed after update it works.

@mdaniel it is actually working, I deleted the current minikube VM, upgrade it and did the exact same steps above and my deployment using the local registry just worked!!

edited Jul 27 '20 at 14:45

Jakub

375
1
9

answered Jul 24 '20 at 15:54

mdaniel

2,561
1
9
13

thanks @mdaniel I don;t there is a need to manually change the daemo.json, I think I'm almost there just missing a small thing. – bachr Jul 24 '20 at 18:13
@bachr is that a statement or a question? I don't understand what your comment means – mdaniel Jul 24 '20 at 20:21
oh it was just a comment I feel like it should be handled by the minikube cli and I should not have to ssh into minikube machine and add a configuration to /etc/docker/daemon.json. – bachr Jul 24 '20 at 21:34
1

docker wants all requests to go over TLS (same as kuberetes), so you have to go out of your way to tell it **not** to use TLS (same as kubernetes); thus, since the way you want to use docker is not the norm, you have to configure it to tell it you prefer an abnormal registry setup – mdaniel Jul 25 '20 at 03:11
I see that makes sense, but wonder if minikube can be configured to start secure registries this way I won't have to alter k8s/docker behavior. – bachr Jul 25 '20 at 16:49

Minikube Failed to pull image http: server gave HTTP response to HTTPS client

1 Answers1