1

I have a kafka streams App which runs on a pod in Kube cluster.

Trying to find a way for Kube to scale up/down the pods based on kafka topic lag.

Has anyone walked the path before and willing to share some details ?.

BMW
  • 42,880
  • 12
  • 99
  • 116
Bala.vrad
  • 53
  • 9
  • 1
    https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ – BMW May 06 '20 at 00:30
  • 1
    You need to consider the number of partitions of your topic which you are consuming. as if you have X partition then you can have X consumers consuming at the same time and scaling up for more than X consumers (pods) will not be useful – Mahmoud Eltayeb May 06 '20 at 00:33

1 Answers1

1

For Scaling up based on the lag, your app should be able to publish metrics (Using micrometer for example) use meter type (Gauge https://micrometer.io/docs/concepts#_gauges)

Then you will need to collect the metrics using monitoring system (prometheus for example)

scale up/down based on the value of the meter/metric.

(Check this sample project as example https://github.com/learnk8s/spring-boot-k8s-hpa)

Note: But consider my comment as the max number of allowed consumers >= number of topic partitions

Mahmoud Eltayeb
  • 436
  • 5
  • 7