Linkerd loadbalance gRPC services in unexpected manner

Question

I'm trying to solve gRPC load balance problem with linkerd, but requests will be evenly distributed only when all services are deployed on same node. If I deploy servers on different node, all request will be directed to one of them.

https://i.stack.imgur.com/OYqNX.jpg

Both gRPC service and client are .Net application. Kubernetes version is v1.12.3. Linkerd version is stable-2.1.0.

Here is the configuration of my gRPC service:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: demogrpc-deploy
  name: demogrpc
spec:
  replicas: 3
  selector:
    matchLabels:
      app: demogrpc
  template:
    metadata:
      labels:
        app: demogrpc
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - topologyKey: kubernetes.io/hostname
            labelSelector:
              matchLabels:
                app: demogrpc
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      containers:
      - image: 192.168.99.25:30000/demogrpchost:1.0.9
        imagePullPolicy: Always
        name: demogrpc
        env:
        - name: GRPC_HOST
          value: "127.0.0.1"
        - name: SERVICE_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: GRPC_PORT
          value: "8000"
        ports:
        - containerPort: 8000
          name: grpc
      imagePullSecrets:
      - name: kubernetes-registry
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: demogrpc
  name: demogrpc
spec:
  clusterIP: None
  ports:
  - port: 8000
    targetPort: 8000
  selector:
    app: demogrpc

How do I make load balance work when services are deployed on different nodes?

Update:

I inject linkerd to my client and it started to distribute requests now, but one of the service is still ignored by load balancer somehow.

Update:

I scaled up the services to 5 and clients 3 and something interesting happened. All services receive requests now but for each clients, their requests are distributed to 4 of the services.

https://i.stack.imgur.com/YGmQd.jpg

score 1 · Answer 1 · answered Dec 23 '18 at 05:41

To configure Linkerd service mesh, each node will have a Linkerd. There will be no service-to-service direct communication across the nodes, only Linkerd is allowed to talk to another Linkerd. And Linkerd will communicate with services which are created in same node.

In your case, you Kubernetes service will talk to Linkerd, then Linkerd will distribute its request to all other Linkerd. This is how requests are distributed across nodes. Then upstream Linkerd will distributed requests among services on its node.

To get more, please check this: linkerd-as-a-service-mesh

I think the article you pointed out is referring to linkerd1. Wherein linkerd is installed on each node i.e.daemonset. but linkerd2 is employed as a sidecar. I also experience the same thing as described by Leisen Chang. Im also wondering what kind of load balancing algorithm linkerd2 uses under the hood. — Bal Chua, Dec 30 '18 at 03:03

Linkerd loadbalance gRPC services in unexpected manner

1 Answers1