NestJS Kafka Rebalance & Partition Assignment Strategies, what will happen to already running tasks?

Question

Let's imagine we have:

• 1 Kafka broker
• 1 topic, 10 partitions
• 1 NestJS producer
• changing number of NestJS consumers (from 1 to 10)

After a while, when number of consumers changes, Kafka initiates a rebalancing. AFAIK, NestJS has its own implementation of the assignment stategy, but I didn't find it in the official documentation. What will happen to already running tasks? Will they be executed in vain? Or maybe there is some way to handle it? Thanks.

I tried the setup described above on my computer, and it looks like NestJS is waiting for the consumer to process its message, and only after that rebalancing occurs.

Answer 1

NestJS uses KafkaJS, so the documentation would be there and you can instrument a stop/rebalance listener to handle those events.

In my experience, rebalancing does not wait for consumers to complete; rather, the consumer group coordinator will signal for them to be terminated.