Kubernetes is sending traffic to unhealthy pods

Question

I have some setup of 2 pods using StatefulSet behind a nodeport service with AWS ingress and sending traffic to this service through ALB endpoint. Then I patch the StatefulSet to have only 1 pod. I have configured this StatefulSet with readiness probe and lifecycle policy like this to avoid getting traffic on unhealthy pod.

Liveness and Readiness Probe:

        livenessProbe:
          exec:
            command: ["/bin/sh", "-c", "reply=$(curl -s -o /dev/null -w %{http_code} http://127.0.0.1:80/health); if [ \"$reply\" -lt 200 -o \"$reply\" -ge 400 ]; then exit 1; fi; cat /tmp/.health-check"]
          initialDelaySeconds: 30
          timeoutSeconds: 10
        readinessProbe:
          exec:
            command: ["/bin/sh", "-c", "reply=$(curl -s -o /dev/null -w %{http_code} http://127.0.0.1:80/health); if [ \"$reply\" -lt 200 -o \"$reply\" -ge 400 ]; then exit 1; fi; cat /tmp/.health-check"]
          timeoutSeconds: 10
          failureThreshold: 2

Lifecycle Policy:

        lifecycle:
          postStart:
            exec:
              command: ["/bin/touch", "/tmp/.health-check"]
          preStop:
            exec:
              command: ["sh", "-c", "rm -r /tmp/.health-check; sleep 60"]

terminationGracePeriodSeconds

      terminationGracePeriodSeconds: 60

But when I patch StatefulSet to have only pod, one get into Terminating state and it became unhealthy but it still receives the traffic. How to debug this issue.

Some timestamps and actions:

Patched STS: Sun Jun 25 16:23:28 UTC 2023

Pod became unhealthy (checked using kubectl get pods and observed 0/1 in status): Sun Jun 25 16:23:47 UTC 2023

Last request received on Pod-2 (checked on kibana): Sun Jun 25 16:24:38 UTC 2023

This question doesn't seem to be about programming – Vega Jun 25 '23 at 17:51 — Vega, Jun 25 '23 at 17:51

Ali Kahoot · Answer 1 · 2023-06-25T18:06:29.303

0

This isn't desired behavior, when a pod goes to terminating state, the traffic isn't sent to it. You should get the Endpoints to be sure whether it is an active endpoint or not. Run

kubectl get endpoints SERVICE_NAME

or can run continuously

watch kubectl get endpoints SERVICE_NAME

the SERVICE_NAME should be the name of the K8s service that you created. Confirm if the unhealthy pod ip address is there or not.

edited Jun 25 '23 at 18:06

answered Jun 25 '23 at 17:52

Ali Kahoot

3,371
2
22
27

I've already checked this and as soon as pod gets into terminating state, it's IP is removed from endpoints. Is there any other thing I can check – Ankit Kumar Jun 25 '23 at 19:58
if thats the case, then traffic will not be sending to that pod, how have you checked that traffic is sending to that pod? – Ali Kahoot Jun 25 '23 at 21:45
As I have mentioned in the question, we have setup ELK and I've verified that pod logs from Kibana – Ankit Kumar Jun 26 '23 at 02:56

Kubernetes is sending traffic to unhealthy pods

1 Answers1