How kafka balances partitions load?

Question

i faced a question with load balancing in kafka. So, i created a topic with 10 partitions and created 2 consumers. The 10 partitions were divided and assigned to these consumers (5 partitions to the first and 5 to the second) and it works fine. Sometimes first consumer works, sometimes second.

But at one moment we can face a situation when for example second consumer receives a message and it takes time (for example 10 minutes) to handle this message.

So, my question is how kafka will decide to which partition store the message?

Round robin in this case i think is not a good idea, because messages in partitions that are handled by second consumer won't be handled until the second consumer finishes the long work.

Updated!

According to the @Milan Baran answer, the load is balanced on the producer side. But in this case, even if we provide a custom Partitioner realization, it will be the same problem that the message that was stored in the partition which was assigned to the consumer that is doing long-term work, will not be processed until this consumer finishes its long-term work.

May be, there are additional load balancer somewhere else?

Why don't you just run more consumers? If you start 10 consumers then when one is busy for a long time you still have 9 more processing the data in the other 9 partitions. — Hans Jespersen, Nov 12 '16 at 18:00
But all messages in 10's partition are not processed until it is busy. Thats the problem — D. Krauchanka, Nov 13 '16 at 21:59
Can you write a custom partitioner that puts the long running messages into a dedicated partition so that all the small ones can be partitioned and processes in the remaining partitions? — Hans Jespersen, Nov 13 '16 at 22:39

score 5 · Answer 1 · answered Nov 11 '16 at 14:28

5

The decision which partition should be used is not up to kafka, but the producer sending a message have to decide. Look at https://kafka.apache.org/documentation#producerconfigs

You can provide a partitioner class to decide which partition to pick.

partitioner.class
Partitioner class that implements the Partitioner interface. org.apache.kafka.clients.producer.internals.DefaultPartitioner

There is a description of the DefaultPartitioner strategy

/**
 * The default partitioning strategy:
 * <ul>
 * <li>If a partition is specified in the record, use it
 * <li>If no partition is specified but a key is present choose a partition based on a hash of the key
 * <li>If no partition or key is present choose a partition in a round-robin fashion
 */

answered Nov 11 '16 at 14:28

Milan Baran

4,133
2
32
49

You answer is good, but i think that there are some additional place where load is balanced. According to your answer it seems that there are no way to process the message that was stored in the partition which was assigned to the consumer that is doing long-term work. – D. Krauchanka Nov 11 '16 at 15:48
What did you mean long-term work? Did you set different group.id for consumers? Or you process messages in same thread as consumer reads. For example you would receive fast_msg, fast_msg, fast_msg, slow_msg, fast_msg and you get stucked on slow_msg and can't get next fast_msg? If this is the case use parallelism to process these messages or a akka-streams. – Milan Baran Nov 12 '16 at 11:01
Using different threads for receiving and processing messages will cause additional problem: it i have slow_msg fast_msg1 fast_msg2 fast_msg3, i can face the situation when slow_msg are still processing but fast_msg1 and fast_msg2 are successfully processed and then the jvm crashes. After the restart my consumer will be pointed at fast_msg3, and slow_msg will be lost. – D. Krauchanka Nov 13 '16 at 22:07
But that's an application problem not kafka one. You can turn off auto commit and manually commit offset when you are sure about topic consistency. But yes, these slow messages are your bottle neck and its hard to get rid of them. What's the point they are so slow? – Milan Baran Nov 14 '16 at 09:30

score 3 · Answer 2 · answered Nov 12 '16 at 02:58

It seems what you need is a QUEUE. ONE partition is consumed by MULTIPLE consumers. Each consumer fetches a record from the partition, process it, and fetch another one. If one consumer takes too much time to process the record, others can still fetch (different) records from the partition.

However, Kafka does NOT support this. Each partition can only be consumed by exactly one consumer in a consumer group.

In a word, you need something else do achieve the goal, such as RabbitMQ.

score 3 · Accepted Answer · answered Nov 22 '16 at 19:57

Thanks everybody for help. But i found an answer for my question. So first of all, there are at least 3 places where kafka balances load:

To assign partitions to consumers "Round robin" or "Range" algorithms are used. This can be configured by setting the partition.assignment.strategy property. By default range is used.
At producer level can be applied a strategy for selecting partition to store message. It can be done by partitioner.class
And the answer for my question. If one consumer processes message for a long time, kafka thinks that this consumer is dead and reassign partitions between another consumers. So when a long time job is done by the consumer, no partitions are assigned to it. When the consumer finishes long time work partitions will be assigned to it again. And no messages will be pending.

How kafka balances partitions load?

3 Answers3