Why access to the k8s service takes too long?

Question

I've created a simple k8s cluster and deployed a microsocks server with the help of kallqvist/microsocks. After exposing it via service, it takes too long for a curl command to connect to this service and retrieve the info.

Prerequisites:

last version of Kubernetes
calico for cni plugin

Steps to reproduce the problem:

Here is a simple command to run this pod inside your cluster:

k run socks --namespace testns --rm -it --image=kallqvist/microsocks:latest --command -- microsocks -1 -p 1080 -u suser -P spassword

// take a look:
# k -n testns get pods -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP              NODE      NOMINATED NODE   READINESS GATES
socks   1/1     Running   0          61m   10.244.225.48   node-fi   <none>           <none>

After that, you can create a simple service (NodePort Service) to expose my pod, here is the command:

k -n testns expose pods/socks --type NodePort --port 1080

Get the service node port by this command:

# k -n testns get svc
NAME    TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
socks   NodePort   10.99.188.25   <none>        1080:31410/TCP   50m

At this moment you can access the service, but it takes approximately 3 minutes. You can test it with this curl command:

curl -x socks5://suser:spassword@<your-server-ip>:31410 http://ifconfig.ovh

Here is my tcpdump packets:

2286    3.279178    5.125.188.197   xxx.xxx.xxx.105 TCP 76  14767 → 31410 [SYN] Seq=0 Win=64240 Len=0 MSS=1400 SACK_PERM=1 TSval=2144430994 TSecr=0 WS=128
2289    3.279248    xxx.xxx.xxx.105 10.244.225.48   TCP 76  44194 → 1080 [SYN] Seq=0 Win=64240 Len=0 MSS=1400 SACK_PERM=1 TSval=2144430994 TSecr=0 WS=128
2292    3.279289    10.244.225.48   xxx.xxx.xxx.105 TCP 76  1080 → 44194 [SYN, ACK] Seq=0 Ack=1 Win=64260 Len=0 MSS=1440 SACK_PERM=1 TSval=3395052955 TSecr=2144430994 WS=128
2293    3.279301    xxx.xxx.xxx.105 5.125.188.197   TCP 76  31410 → 14767 [SYN, ACK] Seq=0 Ack=1 Win=64260 Len=0 MSS=1440 SACK_PERM=1 TSval=3395052955 TSecr=2144430994 WS=128
2500    3.413135    5.125.188.197   xxx.xxx.xxx.105 TCP 68  14767 → 31410 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=2144431131 TSecr=3395052955
2501    3.413148    xxx.xxx.xxx.105 10.244.225.48   TCP 68  44194 → 1080 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=2144431131 TSecr=3395052955
155590  229.785623  5.125.188.197   xxx.xxx.xxx.105 TCP 73  14767 → 31410 [PSH, ACK] Seq=1 Ack=1 Win=64256 Len=5 TSval=2144657489 TSecr=3395052955
155593  229.785729  xxx.xxx.xxx.105 10.244.225.48   Socks   73  Version: 5
155596  229.785792  10.244.225.48   xxx.xxx.xxx.105 TCP 68  1080 → 44194 [ACK] Seq=1 Ack=6 Win=64256 Len=0 TSval=3395279461 TSecr=2144657489
155597  229.785811  xxx.xxx.xxx.105 5.125.188.197   TCP 68  31410 → 14767 [ACK] Seq=1 Ack=6 Win=64256 Len=0 TSval=3395279461 TSecr=2144657489
155598  229.785871  10.244.225.48   xxx.xxx.xxx.105 Socks   70  Version: 5
155599  229.785928  xxx.xxx.xxx.105 5.125.188.197   TCP 70  31410 → 14767 [PSH, ACK] Seq=1 Ack=6 Win=64256 Len=2 TSval=3395279461 TSecr=2144657489
155654  229.922585  5.125.188.197   xxx.xxx.xxx.105 TCP 68  14767 → 31410 [ACK] Seq=6 Ack=3 Win=64256 Len=0 TSval=2144657639 TSecr=3395279461
155655  229.922621  xxx.xxx.xxx.105 10.244.225.48   TCP 68  44194 → 1080 [ACK] Seq=6 Ack=3 Win=64256 Len=0 TSval=2144657639 TSecr=3395279461
155743  230.314622  5.125.188.197   xxx.xxx.xxx.105 TCP 78  14767 → 31410 [PSH, ACK] Seq=6 Ack=3 Win=64256 Len=10 TSval=2144658027 TSecr=3395279461
155744  230.314659  xxx.xxx.xxx.105 10.244.225.48   Socks   78  Version: 5
155745  230.314708  10.244.225.48   xxx.xxx.xxx.105 TCP 68  1080 → 44194 [ACK] Seq=3 Ack=16 Win=64256 Len=0 TSval=3395279990 TSecr=2144658027
155746  230.314732  xxx.xxx.xxx.105 5.125.188.197   TCP 68  31410 → 14767 [ACK] Seq=3 Ack=16 Win=64256 Len=0 TSval=3395279990 TSecr=2144658027
155747  230.314864  10.244.225.48   213.186.33.50   TCP 76  53530 → 80 [SYN] Seq=0 Win=64800 Len=0 MSS=1440 SACK_PERM=1 TSval=2621910285 TSecr=0 WS=128
155772  230.343254  213.186.33.50   10.244.225.48   TCP 76  80 → 53530 [SYN, ACK] Seq=0 Ack=1 Win=17520 Len=0 MSS=1460 SACK_PERM=1 WS=4096 TSval=23 TSecr=2621910285
155773  230.343307  10.244.225.48   213.186.33.50   TCP 68  53530 → 80 [ACK] Seq=1 Ack=1 Win=64896 Len=0 TSval=2621910314 TSecr=23
155777  230.343418  10.244.225.48   xxx.xxx.xxx.105 Socks   78  Version: 5
155778  230.343454  xxx.xxx.xxx.105 5.125.188.197   TCP 78  31410 → 14767 [PSH, ACK] Seq=3 Ack=16 Win=64256 Len=10 TSval=3395280019 TSecr=2144658027
155945  230.472819  5.125.188.197   xxx.xxx.xxx.105 TCP 68  14767 → 31410 [ACK] Seq=16 Ack=13 Win=64256 Len=0 TSval=2144658198 TSecr=3395280019
155946  230.472819  5.125.188.197   xxx.xxx.xxx.105 HTTP    144 GET / HTTP/1.1 
155947  230.472847  xxx.xxx.xxx.105 10.244.225.48   TCP 68  44194 → 1080 [ACK] Seq=16 Ack=13 Win=64256 Len=0 TSval=2144658198 TSecr=3395280019
155948  230.472857  xxx.xxx.xxx.105 10.244.225.48   HTTP    144 GET / HTTP/1.1 
155949  230.472952  10.244.225.48   xxx.xxx.xxx.105 TCP 68  1080 → 44194 [ACK] Seq=13 Ack=92 Win=64256 Len=0 TSval=3395280148 TSecr=2144658199
155950  230.472973  xxx.xxx.xxx.105 5.125.188.197   TCP 68  31410 → 14767 [ACK] Seq=13 Ack=92 Win=64256 Len=0 TSval=3395280148 TSecr=2144658199
155951  230.472984  10.244.225.48   213.186.33.50   HTTP    144 GET / HTTP/1.1 
155958  230.503135  213.186.33.50   10.244.225.48   HTTP    370 HTTP/1.1 200 OK  (text/html)
155959  230.503178  10.244.225.48   213.186.33.50   TCP 68  53530 → 80 [ACK] Seq=77 Ack=303 Win=64640 Len=0 TSval=2621910474 TSecr=56
155961  230.503197  10.244.225.48   xxx.xxx.xxx.105 HTTP    370 HTTP/1.1 200 OK  (text/html)
155962  230.503210  xxx.xxx.xxx.105 5.125.188.197   HTTP    370 HTTP/1.1 200 OK  (text/html)
156054  230.631555  5.125.188.197   xxx.xxx.xxx.105 TCP 68  14767 → 31410 [ACK] Seq=92 Ack=315 Win=64128 Len=0 TSval=2144658357 TSecr=3395280179
156055  230.631555  5.125.188.197   xxx.xxx.xxx.105 TCP 68  14767 → 31410 [FIN, ACK] Seq=92 Ack=315 Win=64128 Len=0 TSval=2144658357 TSecr=3395280179
156056  230.631589  xxx.xxx.xxx.105 10.244.225.48   TCP 68  44194 → 1080 [ACK] Seq=92 Ack=315 Win=64128 Len=0 TSval=2144658357 TSecr=3395280179
156057  230.631599  xxx.xxx.xxx.105 10.244.225.48   TCP 68  44194 → 1080 [FIN, ACK] Seq=92 Ack=315 Win=64128 Len=0 TSval=2144658357 TSecr=3395280179
156058  230.631719  10.244.225.48   213.186.33.50   TCP 68  53530 → 80 [FIN, ACK] Seq=77 Ack=303 Win=64640 Len=0 TSval=2621910602 TSecr=56
156060  230.631739  10.244.225.48   xxx.xxx.xxx.105 TCP 68  1080 → 44194 [FIN, ACK] Seq=315 Ack=93 Win=64256 Len=0 TSval=3395280307 TSecr=2144658357
156061  230.631745  xxx.xxx.xxx.105 5.125.188.197   TCP 68  31410 → 14767 [FIN, ACK] Seq=315 Ack=93 Win=64256 Len=0 TSval=3395280307 TSecr=2144658357
156063  230.660167  213.186.33.50   10.244.225.48   TCP 68  80 → 53530 [FIN, ACK] Seq=303 Ack=78 Win=2097152 Len=0 TSval=95 TSecr=2621910602
156064  230.660191  10.244.225.48   213.186.33.50   TCP 68  53530 → 80 [ACK] Seq=78 Ack=304 Win=64640 Len=0 TSval=2621910631 TSecr=95
156107  230.766792  5.125.188.197   xxx.xxx.xxx.105 TCP 68  14767 → 31410 [ACK] Seq=93 Ack=316 Win=64128 Len=0 TSval=2144658494 TSecr=3395280307
156108  230.766838  xxx.xxx.xxx.105 10.244.225.48   TCP 68  44194 → 1080 [ACK] Seq=93 Ack=316 Win=64128 Len=0 TSval=2144658494 TSecr=3395280307

I have tested it in a GKE cluster, and once the pod is exposed as a LoadBalancer service the curl responds very shortly after. Maybe the processing capacity of the system you have your cluster in has something to do with the issue. Can you try another system with more capacity and compare? — Gabriel Robledo Ahumada, Jun 23 '22 at 22:06
Don't use the LoadBalancer type, expose it through NodePort please @GabrielRobledoAhumada — mostafa8026, Jun 24 '22 at 11:00
i tried to reproduce the issue on my onprem cluster , curl is showing `curl: (7) Can't complete SOCKS5 connection to 0.0.0.0:0. (1) ` — confused genius, Jun 26 '22 at 03:28
@confusedgenius, Could you provide the curl command you've used? — mostafa8026, Jun 26 '22 at 04:35
curl -x socks5://suser:spassword@10.157.160.162:31654 http://ifconfig.ovh > curl: (7) Can't complete SOCKS5 connection to 0.0.0.0:0. (1) — confused genius, Jun 26 '22 at 06:16
Does it take a minute or two to show you this output? If yes, it is similar to my problem because sometimes it doesn't show any response because of timeout. — mostafa8026, Jun 26 '22 at 06:35
I have tested with a NodePort service as well, in a GKE cluster with calico enabled and do not see such behavior. — Gabriel Robledo Ahumada, Jul 01 '22 at 20:45
@GabrielRobledoAhumada, Thanks. I think it is somehow related to my k8s config or something else. I don't know where to start troubleshooting :( — mostafa8026, Jul 02 '22 at 02:38
@mostafa8026 a good troubleshooting start would be to test your deployment in another cluster hosted in a different system. — Gabriel Robledo Ahumada, Jul 04 '22 at 15:24
@GabrielRobledoAhumada, Could you please tell me you k8s version? mine is: `Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:26:19Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:18:48Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"} ` — mostafa8026, Jul 04 '22 at 16:28

CodeWizard · Answer 1 · 2022-06-26T23:39:43.597

Try this out and it might solve your issue

In your service change the externalTrafficPolicy value.
My guess is that you have the default value which is cluster, change it to local and update if it's solved your issue.

### externalTrafficPolicy: Local

apiVersion: v1
kind: Service
spec:
  ports:
    - name: port-8080
      protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 32600
  selector:
    app: my-service
  externalTrafficPolicy: Local
  internalTrafficPolicy: Cluster

externalTrafficPolicy denotes if this Service desires to route external traffic to node-local or cluster-wide endpoints.

"Local" preserves the client source IP and avoids a second hop for LoadBalancer and NodePort type services, but risks potentially imbalanced traffic spreading.
"Cluster" obscures the client source IP and may cause a second hop to another node, but should have good overall load-spreading.

Calico defaults:

https://projectcalico.docs.tigera.io/security/services-cluster-ips

Thanks, but it is not the solution. Everything is the same as before :( — mostafa8026, Jun 27 '22 at 06:04

Why access to the k8s service takes too long?

1 Answers1

Calico defaults: