Unable to connect to the server: net/http: TLS handshake timeout

Question

On minikube for windows I created a deployment on the kubernetes cluster, then I tried to scale it by changing replicas from 1 to 2, and after that kubectl hangs and my disk usage is 100%. I only have one container in my deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: first-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      run: app
  template:
    metadata:
      labels:
        run: app
    spec:
      containers:
      - name: demo
        image: ner_app
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 5000

all I did was run this after the pods were successfully deployed and running

kubectl scale --replicas=2 deployment first-deployment

In another terminal I was watching the pods using

kubectl get pods --watch

But everything is unresponsive and I'm not sure how to recover from this.

When I run kubectl get pods again it gives the following message

PS D:\docker\ner> kubectl get pods
Unable to connect to the server: net/http: TLS handshake timeout

Is there a way to recover, or cancel whatever process is running?

Also my VM's are on Hyper-V for Windows 10 Pro (minikube and Docker Desktop) both have the default RAM allocated - 2048MB

The container in my pod is a machine learning process and the model it loads could be large, in the order of 200MB to 300MB

Closing Docker Desktop and all its related processes solved it for me. — MasterPiece, Apr 22 '23 at 16:57

score 46 · Answer 1 · answered May 15 '20 at 11:17

46

You may have some proxy problems. Try following commands:

$ unset http_proxy
$ unset https_proxy

and repeat your kubectl call.

answered May 15 '20 at 11:17

Daniel Perník

5,464
2
38
46

3

Or add the node address to NO_PROXY environment variable. For instance, export NO_PROXY=$NO_PROXY,192.168.126.139:6443 – Muzammil Dec 08 '20 at 16:56
solved with no_proxy in my case. thanks a lot. – ironrainbow Jun 14 '21 at 02:41
3

I searched the issue, found this answer. Upvoted. Then forgot. Six months later, got the same issue, searched again, found this answer. Wish I could upvote again :-) – Muzammil Jun 24 '21 at 16:02
3

`$ unset all_proxy` is another one to add to the list as a commonly used env var. these are also sometimes all in caps, so `$ unset ALL_PROXY` and same for `HTTP_PROXY`, `HTTPS_PROXY`, `FTP_PROXY` etc – agilgur5 Jul 29 '21 at 21:21
Any alternative for windows powershell ? – Naveen Gopalakrishna Jul 14 '22 at 06:57
For anyone having similar problem for AWS EKS (after adding vpc-cni addon) NO_PROXY=$NO_PROXY;eks.amazonaws.com works. – Gorkem Oct 31 '22 at 10:32
Didn't work for me. I have Mac + intel. Docker desktop Minikube running, trying to get kubernetes dashboard working. – bhosleviraj Aug 06 '23 at 06:34

Dror Harari · Answer 2 · 2023-07-09T09:06:24.387

15

Just happened to me on a new Windows 10 install with Ubuntu distro in WSL2. I solved the problem by running:

$ sudo ifconfig eth0 mtu 1350

or for modern Linux distros that deprecated ifconfig:

$ sudo ip link set dev eth0 mtu 1350

(BTW, I was on a VPN connection when trying the 'kubectl get pods' command)

edited Jul 09 '23 at 09:06

answered Apr 03 '22 at 07:03

Dror Harari

3,076
2
27
25

Thanks! I forgot about this! I was using Wireguard on Windows + WSL2. If there is any network problem usually it was MTU issue. – MichalTheDweller Aug 22 '22 at 11:44
we had the same issue Windows + WSL2, Over wireguard we decreased the MTU to 1400. Wireguard Server was hosted at cloud provider. – igiannak Sep 14 '22 at 15:32
Changing the MTU did the trick for me. Could you please elaborate on how MTU affects this? – Koshur Jan 25 '23 at 08:11
@Koshur - I haven't debugged this but usually this happens when the "Do-not-fragment" bit is set on a packet that is routed through a network whose MTU is set to a lower number than the packet size. – Dror Harari Mar 04 '23 at 02:53
It seems that ifconfig is deprecated. This is an alternative command: sudo ip link set dev eth0 mtu 1350 – Ecyrb Jun 02 '23 at 15:58
This helped me :) Thank you – Josef Veselý Jul 14 '23 at 06:27

Venryx · Answer 3 · 2022-01-03T19:20:18.493

For me, the problem is that Docker ran out of memory. (EDIT: Possibly anyway; I wrote this post a while ago, and am now not so sure that is the root case, but did not write down my rationale, so idk.)

Anyway, to fix:

Fully close your k8s emulator. (docker desktop, minikube, etc.)
Shutdown WSL2. (wsl --shutdown) [EDIT: This step is apparently not necessary -- at least not always, since this time I skipped it, and the problem still resolved.]
Restart your k8s emulator.
Rerun the commands you wanted.

Sometimes it also works to simply:

Right click the Docker Desktop tray-icon, press "Restart Docker", and wait a few minutes for things to restart. (sometimes this fails, with Docker Desktop saying "Docker failed to start", so I'd generally recommend the more thorough process above)

In my case, I also had to increase WSL's memory – David Gomes Mar 26 '22 at 11:36 — David Gomes, Mar 26 '22 at 11:36

score 6 · Answer 4 · edited May 22 '20 at 16:42

6

You can set up resource limits on deployments so that pods will not use the entire available resource in the node.

edited May 22 '20 at 16:42

rphv

5,409
3
29
47

answered Jan 29 '20 at 01:26

ffran09

887
7
9

1

Ok I will try that. How can I recover from the current situation? I came back to my workstation this morning and its still in the same state. My disk is still 100% usage and kubectl is unresponsive. – erotavlas Jan 29 '20 at 14:50
1

I think you should stop minikube and start it again. Does answers in https://stackoverflow.com/questions/56327843/minikube-is-slow-and-unresponsive help you? – ffran09 Jan 29 '20 at 19:28

score 5 · Answer 5 · answered Aug 04 '20 at 11:19

In my case I have my private EKS cluster and there is no 443(HTTPS) enabled in security groups.

My issue is solved after enabling the (HTTPS)443 port in security groups.

Kindly refer for AWS documentation for more details: "You must ensure that your Amazon EKS control plane security group contains rules to allow ingress traffic on port 443 from your connected network"

Abdalrhman Alkraien · Answer 6 · 2022-01-23T13:50:11.397

4

i solved this problem when execute the following command

minikube delete

and then start it

minikube start --vm-driver="virtualbox"

if use this why your pods will deleted

and when run kubectl get pods

you can see this result No resources found in default namespace.

edited Jan 23 '22 at 13:50

answered Jan 23 '22 at 08:54

Abdalrhman Alkraien

645
1
7
22

Chris Halcrow · Answer 7 · 2021-08-24T06:38:04.220

3

You could try $ unset all_proxy to reset the socket proxy.

Also, if you're connected to a VPN, try disconnecting - it seems that can interfere with connecting to a cluster.

edited Aug 24 '21 at 06:38

answered Aug 24 '21 at 03:33

Chris Halcrow

28,994
18
176
206

I had tailscale running, stopping it solved it for me – fredrik Oct 05 '22 at 15:21

score 2 · Answer 8 · answered Mar 23 '22 at 19:15

2

Simply closing cmd, opening again, then

minikube start

And then executing the commands again solved this issue for me. P.S: minikube start took less than a minute

answered Mar 23 '22 at 19:15

Raman Mishra

61
3

score 2 · Answer 9 · answered Apr 23 '23 at 01:50

2

I got the same problem. Usually, this works:

$ unset http_proxy
$ unset https_proxy

And when this⬆️ didn't work today, I just restart my docker and minikube, and it works.

answered Apr 23 '23 at 01:50

levi

39
3

Ri1a · Answer 10 · 2021-10-22T11:01:49.277

I think the other answers don't really mention or refer to the vpn and proxy documentation for minikube: https://minikube.sigs.k8s.io/docs/handbook/vpn_and_proxy/

The NO_PROXY variable here is important: Without setting it, minikube may not be able to access resources within the VM. minikube uses two IP ranges, which should not go through the proxy:

192.168.99.0/24: Used by the minikube VM. Configurable for some hypervisors via --host-only-cidr
192.168.39.0/24: Used by the minikube kvm2 driver.
192.168.49.0/24: Used by the minikube docker driver’s first cluster.
10.96.0.0/12: Used by service cluster IP’s. Configurable via --service-cluster-ip-rang

So adding those IP ranges to your NO_PROXY environment variable should fix the issue.

score 1 · Answer 11 · answered May 02 '22 at 22:58

1

Adding the IP address to the no_proxy list worked for me.

Obtain the IP address from ip addr output.

export no_proxy=localhost,127.0.0.1,<IP_ADDRESS>

answered May 02 '22 at 22:58

Arun

19,750
10
51
60

For those having issue with AWS EKS, NO_PROXY solves problem NO_PROXY=$NO_PROXY;eks.amazonaws.com – Gorkem Oct 31 '22 at 10:33

score 1 · Answer 12 · answered Nov 20 '22 at 18:15

1

restart minikube will work.

But if you don't want to delete it

then you can just switch to other cluster and then switch back.

I just click other kubenete cluster (ex: docker-desktop)

and then click back to the cluster I want to run (ex: minikube)

answered Nov 20 '22 at 18:15

JHIH-LEI

104
6

user4894254 · Answer 13 · 2022-09-20T05:57:44.357

0

If you're on Linux or Mac, go to your virtualbox, and then on the toolbar choose 'Global Tools', then if you see two machines are using the same ip address, you should remove one of them. this image shows virtual box GUI

edited Sep 20 '22 at 05:57

answered Sep 20 '22 at 05:57

user4894254

1
2

score 0 · Answer 14 · answered Oct 31 '22 at 10:38

As this answer comes first on search for net-http-tls-handshake-timeout error

For those having issue with AWS EKS (and likely any K8s), NO_PROXY solves problem by adding related IP/host to environment variable. As suggested in comments for first answer.

For AWS EKS (when seeing this intermittently after vpc-cni addon upgrade) replace for specific region or single url for your use case.

NO_PROXY=$NO_PROXY;eks.amazonaws.com

René · Answer 15 · 2022-12-30T20:07:46.683

0

At least for Windows 10 and 11

$PS C:\oc rollback dc/my-app

Unable to connect to the server: net/http: TLS handshake timeout

For OpenShift 4.x the problem is that for some reason you are logged-out:

$PS C:\oc status

error: You must be logged in to the server (Unauthorized)

logging in by e.g.

$oc login -u developer

resolves the problem

edited Dec 30 '22 at 20:07

answered Dec 28 '22 at 21:13

René

31
3

score 0 · Answer 16 · answered Feb 17 '23 at 12:44

0

Open PowerShell as an administrator and run the command "wsl --shutdown". You will see the same notification in your open Ubuntu terminal.

Open Docker Desktop.

Open a new terminal.

Run the command "minikube status" in the Ubuntu terminal.

Run the Minikube container. You can do this in Docker Desktop.

Run the command "minikube start".

That's it! You don't need to close your computer after this, and Minikube should work fine.

answered Feb 17 '23 at 12:44

Samet Arslantürk

11
1

And please control your Windows Subsystem for Linux. 1- Press Windows key+R to open Run dialog. type: optionalfeatures.exe and hit Enter 2- Scroll to the bottom and uncheck Windows Subsystem for Linux. Click OK. – Samet Arslantürk Feb 20 '23 at 06:22

score 0 · Answer 17 · answered Feb 27 '23 at 14:47

0

I am running my cluster on virtualbox with two external etcd vm clusters and I had to start both etcd VMs. I was able to run kubectl fine afterwards.

answered Feb 27 '23 at 14:47

black swan

1
1

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Mar 10 '23 at 19:23

score 0 · Answer 18 · answered Jul 06 '23 at 09:04

0

I met the same problem and I fix it by sync the system time of the node.

answered Jul 06 '23 at 09:04

Vox

506
2
13

Unable to connect to the server: net/http: TLS handshake timeout

18 Answers18

Linked