Kubernetes service does not distribute requests between pods

Question

a have a problem with Kubernetes service. My service only sends requests to one pod ignoring other pods. I don't know why and how can I debug it. It should distribute request in a round-robin way. For me, it seems that something's wrong service but I don't know to debug it. Outputs of kubectl describe service and nodes along with endpoints


apiVersion: v1
kind: Service
metadata:
  name: web-svc
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30002
  selector:
    app: web
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  labels:
    app: web
spec:
  selector:
    matchLabels:
      app: web
  replicas: 3
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web-app
        image: webimage
        ports:
          - containerPort: 80
        imagePullPolicy: Never
        resources:
          limits:
            cpu: "0.5"
          requests:
            cpu: "0.5"


Name:                     web-svc
Namespace:                default
Labels:                   <none>
Annotations:              Selector:  app=webpod
Type:                     NodePort
IP:                       10.111.23.112
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  30002/TCP
Endpoints:                10.244.1.7:80,10.244.1.8:80,10.244.1.9:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>


Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP:                10.96.0.1
Port:              https  443/TCP
TargetPort:        6443/TCP
Endpoints:         172.18.0.3:6443
Session Affinity:  None
Events:            <none>

Name:         web-depl-5c87b748f-kvtqr
Namespace:    default
Priority:     0
Node:         kind-worker/172.18.0.2
Start Time:   Mon, 04 May 2020 04:20:34 +0000
Labels:       app=webpod
              pod-template-hash=5c87b748f
Annotations:  <none>
Status:       Running
IP:           10.244.1.8
IPs:
  IP:           10.244.1.8
Controlled By:  ReplicaSet/web-depl-5c87b748f
Containers:
  web:
    Container ID:   containerd://8b431d80fd729c8b0d7e16fa898ad860d1a223b3e191367a68e3b65e330fe61a
    Image:          web
    Image ID:       sha256:16a4c5d1a652b1accbacc75807abc1d9a05e2be38115dc8a5f369a04a439fad2
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 04 May 2020 04:20:36 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:  500m
    Requests:
      cpu:        500m
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-c9tgf (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-c9tgf:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-c9tgf
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>
=========
Name:         iweblens-svc
Namespace:    default
Labels:       <none>
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2020-05-04T04:20:36Z
Subsets:
  Addresses:          10.244.1.7,10.244.1.8,10.244.1.9
  NotReadyAddresses:  <none>
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    <unset>  80    TCP

Events:  <none>

how are you accessing the service ? always using the same node ip or different node ip? how many worker nodes you have on the cluster? — Arghya Sadhu, May 04 '20 at 06:19
How are you accessing the service? Using the node IP, `kubectl port-forward`, something else? With what client tool? How are you verifying that only one pod is getting connections? — David Maze, May 04 '20 at 09:55

score 1 · Answer 1 · answered Feb 23 '23 at 17:35

Is the client using a persistent/long-lived connection? Because the service endpoint will only distribute the new connections in a round-robin manner as per your setting. Kubernetes doesn't offer any built-in mechanism to load balance long-lived connections. For long-lived connections, you can handle the load balancing on the client side or use a reverse proxy (service mesh/traefik ingress) which can take care of the load balancing responsibility.

score 0 · Answer 2 · answered May 04 '20 at 09:39

I strongly recommend debugging your service according to the the official documentation:

An issue that comes up rather frequently for new installations of Kubernetes is that a Service is not working properly. You’ve run your Pods through a Deployment (or other workload controller) and created a Service, but you get no response when you try to access it. This document will hopefully help you to figure out what’s going wrong.

There you will find a step by step guide showing what and how to check each component that could go wrong. For example:

Is the Service defined correctly?
Are the Pods working?
Is the kube-proxy working?

Go through it step by step and please let me know if you find a solution. If the problem would still persist we can dive deeper in order to fix it.

Kubernetes service does not distribute requests between pods

2 Answers2

Linked