Load Distribution not working in Kubernetes, Request is going to single pod

Question

I m using Minikube for kubernetes deployment and Operating system i m using is Ubuntu 18.04

I have a Deployment of my application with 3 replicas. So when i deploy 3 pods gets deployed . when i increase the number of http request all the request are forwarded to the a single pod while there are no logs recorded in the other two replicas.

Any ideas on how the load get distributed across all the pods.

i have the deployment.yaml and respective service.yaml file below

    deployment.yaml


    apiVersion: apps/v1
    kind: Deployment
    metadata:
          name: master
      labels:
       app: master-container
    spec:
     selector:
       matchLabels:
         app: master-container
     replicas: 3
     template:
       metadata:
         labels:
           app: master-container
       spec:
         volumes:
         - name: logs-dir
           hostPath:
            path: /data/logs/
            type: DirectoryOrCreate
         containers:
          - name: master-container
            image: image:1
            volumeMounts:
            - mountPath: /data/workspace/logs/
              name: logs-dir

    service.yaml
    
apiVersion: v1
kind: Service
metadata:
  name: master-container
  labels:
    app: master-container
spec:
  selector:
    app: master-container
  ports:
  - port: 2000
    protocol: TCP
    targetPort: 2000
    name: can-port
    nodePort: 32000
  - port: 2002
    protocol: TCP
    targetPort: 2002
    name: can-ajp-port
    nodePort: 32002
  - port: 2003
    protocol: TCP
    targetPort: 2003
    name: cas-port
    nodePort: 32003
  - port: 2005
    protocol: TCP
    targetPort: 2005
    name: cas-ajp-port
    nodePort: 32005
  - port: 31900
    protocol: TCP
    targetPort: 31900
    name: cas-master-port
    nodePort: 31900
  type: NodePort

Have a look at the below image this is the output of the command

kubectl describe svc canmastercontainer

We can see that service is up and all the ports are accessible . The only issue i m facing is that the load distribution is not happening across the pods

i have tried giving type as NodePort , LoadBalancer and ClusterIp all the three didnt work out . All the http request are getting transferred to the single pod .

Any solutions would be appriciated thank you

Could you provide informations from one test scenario? Your manifests including `app: master-container` but in the screen selector and label is `app: canmasterconainer`. Also type is different between screen and manifests. Are you used helm to deploy this app (saw Helm annotations), if so can you share chart? Did you consider using [Nginx Ingress Patch Matching](https://kubernetes.github.io/ingress-nginx/user-guide/ingress-path-matching/)? Can you see any issues in pods logs? — PjoterS, Nov 19 '20 at 16:42
@slashpai All the pods are working fine and there are no issues in the pod logs . The only issue here is whenever i make a request from the browser the request transferred to a single pod , while there are no logs generated in the other two pods . — Umesh, Nov 20 '20 at 04:05
@PjoterS : All the selector matching in my deployment and all the pods are running. For the helm part i m using helm for deployment and the above files is what i have been using for the deployment . The issue i m facing here is there is no Load Distribution happening b/w the pods even when i have multiple replicas of the pod. when i make a http request from the browser all the request gets transferred to a single pod while the other replicas of the pod remain idle i,e the request is not hitting those pods . i would like to know how the load distribution works . — Umesh, Nov 20 '20 at 04:21
Could you ensure that `kube-proxy` put proper rules to `iptables`? Also check if `iptables rules works as expected? Did you consider using Ingress? — PjoterS, Nov 20 '20 at 12:14

score 1 · Answer 1 · answered Nov 20 '20 at 15:10

1

If you are using port forwarding then the load will not be distributed across all pods (endpoints).

Please refer to this link for more information.

answered Nov 20 '20 at 15:10

Hemanth H L

111
1
6

score 1 · Answer 2 · answered Nov 30 '20 at 15:02

Every node in a Kubernetes cluster runs a kube-proxy. kube-proxy is responsible for implementing a form of virtual IP for Services of type other than ExternalName.

kube-proxy can work in a few modes:

However, expect built-in Kubernetes methods you can use 3rd party software like Nginx Ingress.

Test scenario

Nginx Ingress, bare metal consideration on GCP Ubuntu VM.

Hello world YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-1
spec:
  replicas: 1
  selector:
    matchLabels:
      key: application-1
  template:
    metadata:
      labels:
        key: application-1
    spec:
      containers:
      - name: hello1
        image: gcr.io/google-samples/hello-app:1.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080

Service Yaml:

apiVersion: v1
kind: Service
metadata:
  name: service-one
spec:
  selector:
    key: application-1
  ports:
    - port: 80
      targetPort: 8080

Ingress Yaml:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"      
spec:
  rules:
  - http:
      paths:
      - path: /one
        backend:
          serviceName: service-one
          servicePort: 80
      - path: /two
        backend:
          serviceName: service-two
          servicePort: 80

Ingress Description:

Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /one   service-one:80 (192.168.243.80:8080,192.168.243.84:8080,192.168.243.85:8080)
              /two   service-two:80 (192.168.243.81:8080,192.168.243.82:8080,192.168.243.83:8080)

Tests

$ curl 34.89.244.41:31337/one
Hello, world!
Version: 1.0.0
Hostname: deployment-1-77ddb77d56-4x4pd
$ curl 34.89.244.41:31337/one
Hello, world!
Version: 1.0.0
Hostname: deployment-1-77ddb77d56-vq72x
$ curl 34.89.244.41:31337/one
Hello, world!
Version: 1.0.0
Hostname: deployment-1-77ddb77d56-v5826
$ curl 34.89.244.41:31337/one
Hello, world!
Version: 1.0.0
Hostname: deployment-2-fb984955c-xk5h9
$ curl 34.89.244.41:31337/two
Hello, world!
Version: 2.0.0
Hostname: deployment-2-fb984955c-lw74g
$ curl 34.89.244.41:31337/two
Hello, world!
Version: 2.0.0
Hostname: deployment-2-fb984955c-xk5h9
$ curl 34.89.244.41:31337/two
Hello, world!
Version: 2.0.0
Hostname: deployment-2-fb984955c-lw74g
$ curl 34.89.244.41:31337/two
Hello, world!
Version: 2.0.0
Hostname: deployment-2-fb984955c-8pfls
$ curl 34.89.244.41:31337/two
Hello, world!
Version: 2.0.0
Hostname: deployment-2-fb984955c-8pfls

As you can see, each request was redirected to different pod.

If this won't work, please provide more details which was asked under your question.

Load Distribution not working in Kubernetes, Request is going to single pod

2 Answers2

Test scenario

Tests