68

While using kubectl port-forward function I was able to succeed in port forwarding a local port to a remote port. However it seems that after a few minutes idling the connection is dropped. Not sure why that is so.

Here is the command used to portforward:

kubectl --namespace somenamespace port-forward somepodname 50051:50051

Error message:

Forwarding from 127.0.0.1:50051 -> 50051
Forwarding from [::1]:50051 -> 50051
E1125 17:18:55.723715    9940 portforward.go:178] lost connection to pod

Was hoping to be able to keep the connection up

Stanley
  • 2,798
  • 5
  • 22
  • 44

6 Answers6

45

Setting kube's streaming-connection-idle-timeout to 0 should be a right solution, but if you don't want to change anything, you can use while-do construction

Format: while true; do <<YOUR COMMAND HERE>>; done

So just inputing in CLI: while true; do kubectl --namespace somenamespace port-forward somepodname 50051:50051; done should keep kubectl reconnecting on connection lost

user2563451
  • 779
  • 6
  • 8
  • Is there some solution for windows cmd? – Andrew Sneck Aug 27 '19 at 10:19
  • Sorry @AndrewSneck, have no experience with win console. But as far as I know latest windows has unix cli built-in you should try it out – user2563451 Aug 28 '19 at 11:15
  • 2
    +1 that was the better solution for me: forwarding to kubernetes dashboard from a jumper host, where no kubelet is running – KiteUp Sep 20 '19 at 11:02
  • 2
    @AndrewSneck powershell: while ($true) { kubectl port-forward your-command-here } – Simon Ness May 11 '20 at 11:01
  • 2
    kubectl process doesn't exit even when timeout(got such type of log: E0621 10:11:52.183060 20798 portforward.go:340] error creating error stream for port 9191 -> 9191: Timeout occured). so I doubt this while loop ever loops when timeout. – Lei Yang Jun 21 '21 at 02:32
  • When I receive "lost connection to pod" message, kubectl exits. There may be some merit in a loop. – fiidim Oct 28 '21 at 13:08
  • In my situation the kubectl did not exist when timeout @user2563451 – Dolphin Dec 16 '21 at 04:30
  • I'm having this issue but it didn't help in my case. – titanic Mar 18 '23 at 01:51
39

I solved this by keeping the connection alive, e.g. using curl or nc.

Forward the port:

kubectl --namespace somenamespace port-forward somepodname 50051:50051

In another terminal, keep the connection alive by reaching out to the port every 10 seconds:

while true ; do nc -vz 127.0.0.1 50051 ; sleep 10 ; done
Marcel
  • 983
  • 10
  • 8
  • 2
    This is the only answer that actually keeps the connection alive without changing anything on the server side. Putting the port-forward command in a loop will break long poll connections which is the main reason I have an issue with port-forward timing out in the first place. – voutasaurus Sep 18 '20 at 20:14
  • 3
    This worked! why? God knows ... where are the days of PLAIN C! and unix pizza boxs... – user63898 Jan 07 '21 at 07:41
  • Can you explain why it works, I am very curious.@Marcel – Dolphin Dec 24 '21 at 03:48
  • @Dolphin the port-forward command times out if it isn't used. The second command reaches out to the port and keeps it alive – Marcel Dec 24 '21 at 19:07
  • It didn't help me unfortunately... – titanic Mar 18 '23 at 01:51
35

Seems there is a 5 minute timeout that can be overridden with kubelet parameters:

https://github.com/kubernetes/kubernetes/issues/19231

If you want to pass something higher than 5 minutes (or unlimited) into your kubelets, you can specify the streaming-connection-idle-timeout. E.g. --streaming-connection-idle-timeout=4h to set it to 4 hours. Or: --streaming-connection-idle-timeout=0 to make it unlimited. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)

Mohamed Elqdusy
  • 3
  • 1
  • 3
akauppi
  • 17,018
  • 15
  • 95
  • 120
17

For windows make such bat (God forgive me)

:1
oc port-forward PODNAME 8003:8080
goto 1
Gustly
  • 171
  • 1
  • 3
0

If you are running your Kubernetes cluster behind a load balancer (like HAProxy), it could happen that the timeout configured in kubelet is bigger than the timeout configured in the HAProxy.

For instance, the streamingConnectionIdleTimeout setting in Kubelet by default is 4h:

$ kubectl proxy --port=8001 &
$ NODE_NAME="XXXX"; curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' | grep streaming
  "streamingConnectionIdleTimeout": "4h0m0s",

But if in HAProxy (or your preferred LB) you have these settings:

defaults
  timeout client 1m
  timeout server 1m
...

Trying to execute a port-forwarding will timeout if you don't have any activity over the app:

$ date; kubectl port-forward service/XXXX 1234:80
Mon Jul  5 10:58:20 CEST 2021
Forwarding ...
# after a minute
E0705 10:59:21.217577   64160 portforward.go:233] lost connection to pod

In order to fix this, a solution would be to increase the timeout (be careful with this, because depending on your cluster it can have undesirable effects) or bypass the LB when doing port-forwarding connecting directly to the API server (if your environment allows it).

Pigueiras
  • 18,778
  • 10
  • 64
  • 87
0

Here is a bash function that I use, to bypass port-forward with timeout issues:

function pfpod(){
pod=$1
portloc=$2
portrem=$3
while true
  do
    podname=`kubectl get pods -o name | awk -F'/' '{print $2}'| grep "$pod"| head -1`
    kubectl port-forward $podname $portloc:$portrem
done
}

I have added a kubectl call to match partial podname, in case you have many pods for a deployment, you can just give partial name and the top pod will be used. Otherwise you can also provide the full pod name.

Sample Usage:

pfpod elasticsearch 9201 9200

Vishalendu
  • 73
  • 2
  • 13