0

We currently have a Kafka partition with 3 and 3 Kubernetes pods for each Kafka partition. We observed that there is a performance lag because of less number of pods. We are planning to increase the pods. We observed that lag is happening during specific timing, so we thought of using Kubernetes HPA.

  1. What are the drawbacks if we just increase the partition to 6 and implement Kubernetes HPA
  2. Whats the advantage of using Keda/knative for the same ?

reference : https://www.confluent.io/events/kafka-summit-americas-2021/intelligent-auto-scaling-of-kafka-consumers-with-workload-prediction/, https://itnext.io/kafka-consumer-autoscaling-with-keda-41310f80a62a

Spartan
  • 3,213
  • 6
  • 26
  • 31
  • You can't usefully have more listener replicas than partitions. Do you have a more specific question or a setup you're having trouble with? – David Maze Feb 23 '23 at 15:36
  • we can have more listeners than replicas but not much impact right? since we are planning to increase the pods, should I go with k8s HPA and fixed partition considering auto-scaling? or I have to configure keda/knative for the same ? – Spartan Feb 24 '23 at 06:52
  • KEDA/KNative would be better. Don't scale Kafka consumers on CPU/memory (see my comment below) – OneCricketeer Feb 24 '23 at 23:29

1 Answers1

1

  1. Increasing the partition count will definitely help with throughput. HPA will work on metrics like CPU and memory using which you could still scale. But if you could keep replicas equal to number of partition that will work like charm eg if you have 6 partition have 6 replica always running.

  2. Keda works in event-driven fashion so it will monitor the consumer lag (you could configure that) and it will do the autoscaling better than k8s HPA.

Neenad
  • 861
  • 5
  • 19
  • if no.of messages gets increased, CPU utilization should also go up in the pods and HPA takes place right? what do you mean by no.of.partition == replica count ? you mean considering HPA replicas ? – Spartan Feb 24 '23 at 06:54
  • Yeah I agree that HPA will work with CPU or Memory consumption and that would be enough. If you see that scaling is not great with HPA then move to Keda – Neenad Feb 24 '23 at 07:03
  • If you have 6 partition have 6 replica always running if cost is not a big deal. – Neenad Feb 24 '23 at 07:06
  • cost is the problem as usual :( thanks for your input – Spartan Feb 24 '23 at 08:49
  • CPU scaling is **not** recommended. If you scale up, then the consumer group rebalances, and the CPU immediately falls back down, causing pods to scale down. Then that repeats endlessly. – OneCricketeer Feb 24 '23 at 23:27