0

I have two partitions and messages in avro-format. I send the messages via the Kafka REST proxy API. I use the key for the messages. The key is a string. For example, there are my keys:

41:46-300, 41:45-300, 41:44-300, 41:43-300, 41:42-300.

But the messages are uneven distributed. In the partition 0 there are messages with keys 41:46-300, 41:45-300, 41:44-300, and 41:43-300. And in the partition 1, there are only messages with the key 41:42-300.

Kafka version: 2.4

Why does this happen?

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Demid Demidov
  • 11
  • 4
  • There was a good talk at the last Kafka summit on this topic: https://www.confluent.io/events/kafka-summit-london-2023/the-dark-and-dirty-side-of-fixing-uneven-partitions/ – A.J. Jun 16 '23 at 16:10

1 Answers1

0

Kafka uses Murmur2 hashing to distribute keys, not an evenly distributed round-robin mechanism.

So, this means all events in the same partition ended up with hashes that modulo'd into that partition.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245