1

I am trying find out if it's possible to horizontally scale a strimzi kafka connect cluster to take advantage of pod/node autoscaling as I have a production load that massively fluctuates.

I have tried applying a hpa configuration to the resulting kafka connect deployment but it seems to conflict with kafkaconnect replica definition as new pods start up and are killed very quickly. I assume there is some strimzi mechanism that synchronises the deployment size with the kafkaconnect manifest preventing the hpa from working.

Does anyone know if this is at all possible?

jproxor
  • 13
  • 2

1 Answers1

1

You cannot autoscale the deployment. You have to autoscale the KafkaConnect custom resource which supports the scale subresource and which defines the number of replicas.

Just keep in mind that autoscaling Kafka Connect is not completely trivial:

  • You need to make sure there are enough tasks to be scheduled on the new Pods and that your connectors support using multiple tasks
  • For sink connectors, you have to make sure enough partitions
  • etc.

Soit is not always as easy as adding more pods when CPU is more than 80% utilized or something like that.

Jakub
  • 3,506
  • 12
  • 20
  • Thank you Jakub, that's super helpful. I had incorrectly thought the kafkaconnect didn't support the scale subresource, but it was due to my misconfiguration. – jproxor Jan 26 '22 at 15:45