Kubernetes service routes traffic to only one of 5 pods

Question

i'm playing around with k8s services. I have created simple Spring Boot app, that display it's version number and pod name when curling endpoint:

curl localhost:9000/version
1.3_car-registry-deployment-66684dd8c4-r274b

Then i dockerized it, pushed into my local Kind cluster and deployed with 5 replicas. Next I created service targeting all 5 pods. Lastly, i exposed service like so:

kubectl port-forward svc/car-registry-service 9000:9000

Now when curling my endpoint i expected to see randomly picked pod names, but instead I only get responses from single pod. Moreover, if i kill that one pod then my service stops working, ie i'm getting ERR_EMPTY_RESPONSE, even though there are 4 more pods available. What am I missing? Here's my deployment and service yamls:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: car-registry-deployment
spec:
  replicas: 5
  selector:
    matchLabels:
      app: car-registry
  template:
    metadata:
      name: car-registry
      labels:
        app: car-registry
    spec:
      containers:
        - name: car-registry
          image: car-registry-database:v1.3
          ports:
            - containerPort: 9000
              protocol: TCP
              name: rest
          readinessProbe:
            exec:
              command:
                - sh
                - -c
                - curl http://localhost:9000/healthz | grep "OK"
            initialDelaySeconds: 15
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: car-registry-service
spec:
  type: ClusterIP
  selector:
    app: car-registry
  ports:
    - protocol: TCP
      port: 9000
      targetPort: 9000

can you validate that labels weren't removed from other 4 pods at runtime? — rok, Nov 06 '22 at 11:27
Try this similar [stack question](https://stackoverflow.com/questions/60660070/kubernetes-services-portforwarding-to-all-pods-through-deployment?rq=1) may help to resolve your issue. — Veera Nagireddy, Nov 06 '22 at 14:04

cajual · Answer 1 · 2022-11-06T02:43:00.113

-1

You’re using TCP, so you’re probably using keep-alive. Try to hit it with your browser or a new tty.

Try:

curl -H "Connection: close" http://your-service:port/path

Else, check kube-proxy logs to see if there’s any additional info. Your initial question doesn’t provide much detail.

edited Nov 06 '22 at 02:43

answered Nov 05 '22 at 21:40

cajual

31
3

Tried couple of different terminals and browsers but they still hit one pod :( – brtk Nov 05 '22 at 23:15
I added more context — have you checked any logs in kube-proxy? There’s nothing wrong with your manifests. – cajual Nov 06 '22 at 02:43

Kubernetes service routes traffic to only one of 5 pods

1 Answers1