Minikube does not start, kubectl connection to server was refused

Question

Scouring stack overflow solutions for similar problems did not resolve my issue, so hoping to share what I'm currently experiencing to get help debugging this.

So a small preface; I initially installed minikube/kubectl a couple days back. I went ahead and tried following the minikube tutorial today and am now experiencing issues. I'm following the minikube getting started guide.

I am on MacOS. My versions:

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Unable to connect to the server: net/http: TLS handshake timeout

$ minikube version

minikube version: v0.26.1

$ vboxmanage --version

5.1.20r114629

The following are a string of commands I've tried to check responses..

$ minikube start

Starting VM...
Getting VM IP address...
Moving files into cluster...
E0503 11:08:18.654428   20197 start.go:234] Error updating cluster:  downloading binaries: transferring kubeadm file: &{BaseAsset:{data:[] reader:0xc4200861a8 Length:0 AssetName:/Users/philipyoo/.minikube/cache/v1.10.0/kubeadm TargetDir:/usr/bin TargetName:kubeadm Permissions:0641}}: Error running scp command: sudo scp -t /usr/bin output: : wait: remote command exited without exit status or exit signal

$ minikube status

cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.103

Edit:

I don't know what happened, but checking the status again returned "Misconfigured". I ran the recommended command $ minikube update-context and now the $ minikube ip points to "172.17.0.1". Pinging this IP returns request timeouts, 100% packet loss. Double-checked context and I'm still using "minikube" both for context and cluster:

$ kubectl config get-cluster

$ kubectl config get-context

$ kubectl get pods

The connection to the server 192.168.99.103:8443 was refused - did you specify the right host or port?

Reading github issues, I ran into this one: kubernetes#44665. So...

$ ls /etc/kubernetes

ls: /etc/kubernetes: No such file or directory

Only the last few entries

$ minikube logs

May 03 18:10:48 minikube kubelet[3405]: E0503 18:10:47.933251    3405 event.go:209] Unable to write event: 'Patch https://192.168.99.103:8443/api/v1/namespaces/default/events/minikube.152b315ce3475a80: dial tcp 192.168.99.103:8443: getsockopt: connection refused' (may retry after sleeping)
May 03 18:10:49 minikube kubelet[3405]: E0503 18:10:49.160920    3405 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.99.103:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.99.103:8443: getsockopt: connection refused
May 03 18:10:51 minikube kubelet[3405]: E0503 18:10:51.670344    3405 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.99.103:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0: dial tcp 192.168.99.103:8443: getsockopt: connection refused
May 03 18:10:53 minikube kubelet[3405]: W0503 18:10:53.017289    3405 status_manager.go:459] Failed to get status for pod "kube-controller-manager-minikube_kube-system(c801aa20d5b60df68810fccc384efdd5)": Get https://192.168.99.103:8443/api/v1/namespaces/kube-system/pods/kube-controller-manager-minikube: dial tcp 192.168.99.103:8443: getsockopt: connection refused
May 03 18:10:53 minikube kubelet[3405]: E0503 18:10:52.595134    3405 rkt.go:65] detectRktContainers: listRunningPods failed: rpc error: code = Unavailable desc = grpc: the connection is unavailable

I'm not exactly sure how to ping an https url, but if I ping the ip

$ kube ping 192.168.99.103

PING 192.168.99.103 (192.168.99.103): 56 data bytes
64 bytes from 192.168.99.103: icmp_seq=0 ttl=64 time=4.632 ms
64 bytes from 192.168.99.103: icmp_seq=1 ttl=64 time=0.363 ms
64 bytes from 192.168.99.103: icmp_seq=2 ttl=64 time=0.826 ms
^C
--- 192.168.99.103 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.363/1.940/4.632/1.913 ms

Looking at kube config file... $ cat ~/.kube/config

apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://localhost:6443
  name: docker-for-desktop-cluster
- cluster:
    certificate-authority: /Users/philipyoo/.minikube/ca.crt
    server: https://192.168.99.103:8443
  name: minikube
contexts:
- context:
    cluster: docker-for-desktop-cluster
    user: docker-for-desktop
  name: docker-for-desktop
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: docker-for-desktop
  user:
    client-certificate-data: <removed>
    client-key-data: <removed>
- name: minikube
  user:
    client-certificate: /Users/philipyoo/.minikube/client.crt
    client-key: /Users/philipyoo/.minikube/client.key

And to make sure my key/crts are there:

$ ls ~/.minikube

addons/              ca.pem*              client.key           machines/            proxy-client.key
apiserver.crt        cache/               config/              profiles/
apiserver.key        cert.pem*            files/               proxy-client-ca.crt
ca.crt               certs/               key.pem*             proxy-client-ca.key
ca.key               client.crt           logs/                proxy-client.crt

Any help in debugging is super appreciated!

Does minikube ssh work to get you into the VM? Could try running the command that failed- sudo scp /Users/philipyoo/.minikube/cache/v1.10.0/kubeadm 192.168.99.103:/usr/bin then stop and start. Maybe also ensure that you can sudo without a password in your terminal, and then try stop and start. — Jonah Benton, May 03 '18 at 19:07
Ok, so the ssh key should be present somewhere in your .minikube tree. I'm running an older minikube on linux, and use ssh -o StrictHostKeyChecking=no -i ~/.minikube/machines/minikube/id_rsa docker@192.168.99.100. See if there is a key somewhere, and see if it works for a root user in the VM. minikube startup is strangely unreliable as well for me. It is often hard to tell why something fails, but figuring out how to run the failed command manually, then restarting, often works. — Jonah Benton, May 03 '18 at 19:31
Thank you for the response @JonahBenton , I found an ssh key in the same directory `~/.minikube/machines/minikube/id_rsa` but when trying to ssh with it, I get a timeout, so I'm guessing the key isn't the right one. I don't see any other keys within minikube dir — philip yoo, May 03 '18 at 20:08
Interesting- minikube ssh works, but using that key file with ssh manually does not, and times out? Hmm- something else is likely wrong with the manual ssh, maybe wrong IP? In any event, if you hit a wall, stopping minikube, deleting the .minikube directory, then trying minikube start again is sometimes frustratingly effective. — Jonah Benton, May 03 '18 at 20:21
@JonahBenton haha ya i'm not sure what was wrong, but deleting the dir, removing the vm, and starting fresh seems to have worked. Thanks for your help! — philip yoo, May 03 '18 at 21:17
Great! It's not you, it's minikube, unfortunately, it still needs improvements in reliability. Cheers! — Jonah Benton, May 03 '18 at 21:21

score 25 · Accepted Answer · answered May 03 '18 at 21:22

25

For posterity, the solution to this problem was to delete the

.minikube

directory in the user's home directory, and then try again. Often fixes strange minikube problems.

answered May 03 '18 at 21:22

Jonah Benton

3,598
1
16
27

1

Should we do minikube stop && minikube delete before deleting ~/.minikube or is that not necessary? – user674669 Oct 29 '18 at 21:41
2

yes to minikube stop, you do not want minikube running when deleting its state management out from underneath it. no need to do minikube delete, that deletes part of the .minikube tree, but not the whole thing, which is why it doesn't always fix minikube issues. – Jonah Benton Oct 29 '18 at 21:45

Buddhi · Answer 2 · 2019-04-12T11:47:15.290

I had the same issue when I started minikube.

OS

MacOs HighSierra

Minikube

minikube version: v0.33.1

kubectl version

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:38:32Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:28:14Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}

Solution 1

I just change the permission of the kubeadm file and start the minikube as below. Then it works fine.

sudo chmod 777 /Users/buddhi/.minikube/cache/v1.13.2/kubeadm

In general, you have to do

sudo chmod 777 <PATH_TO_THE_KUBEADM_FILE>

Solution 2

If you no longer need the existing minikube cluster you can try out this.

minikube stop
minikube delete
minikube start

Here you stop and delete existing minikube cluster and create another one.

Hope this might help someone.

Minikube does not start, kubectl connection to server was refused

Edit:

Only the last few entries

2 Answers2