0

I have a worker in spring-boot which listen to kafka topic with 20 partitions. I created the following listener:

@KafkaListener(topics = "mytopic")
public void listen(@Payload(required = true) IncomingMessage msg, 
        @Header(KafkaHeaders.RECEIVED_PARTITION_ID) int partition,
        @Header(KafkaHeaders.OFFSET) long offset) {

In this listener, I print the partition, and found out all my messages are coming from the same partition number - 17. This means the producer side is writing all messages to the same partition.

My container factory is ConcurrentKafkaListenerContainerFactory and so I want to be able to handle multiple events at the same time, this means I need different partitions to also have events in them..

The producer side in a Linux machine is kafkacat, which produce events into this topic. It seems like kafkacat doesn't have the ability to send to partition in a round-robin style.

The problem is, I must use some CLI tool to produce the events and not a service. Is there any way to overcome this issue using CLI tool? I couldn't find any cli tool which does this.

I thought about maintaining a partition-number in a file, and read it, and then provide the partition number in the kafkacat command, and later increment the (partition-number)%partitions, but this is not thread safe.

Note: the topic doesn't have Key, I only produce Value messages, and doesn't care about the Key.

toto
  • 1,197
  • 2
  • 15
  • 26
  • 1
    The default Kafka Producer partitioner is round robin when no key exists in the record. Are you sure the kafkacat utility is not being called with either the -K or -p switch? – Alexandre Juma Mar 04 '20 at 15:07
  • Can you do a: `kafka-consumer-groups.sh --bootstrap-server : --group --describe` PS: If you don't know your consumer group name, just run with --list first – Alexandre Juma Mar 04 '20 at 15:11

2 Answers2

1

As I commented earlier, found this solution at github - just need to add this parameter to kafkacat:

-X topic.partitioner=murmur2_random
toto
  • 1,197
  • 2
  • 15
  • 26
0

As commented, kafkacat should follow the same murmur2 / round robin strategies that any other Kafka client would.

Of course, you're more than welcome to write your own Spring Boot CLI tool to do your own logic

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • That's weird, my bash script is: kafkacat -P -t "$mytopic" -b "$mybroker", when I run it multiple times, I always get the event sent to the same partition (partition 17) I have no idea why – toto Mar 04 '20 at 15:19
  • Maybe open a github issue? See what they think? – OneCricketeer Mar 04 '20 at 15:21
  • I think there is no way for a CLI tool to maintain the last partition number which it was last sent to. Because it's not a service which holds the state, it's just a cli tool. – toto Mar 04 '20 at 15:23
  • CLI tools hold state... just because there isn't a web server or database doesn't mean in memory maps and variables aren't used – OneCricketeer Mar 04 '20 at 15:24
  • It's also open source, so you're welcome to dig around in the code – OneCricketeer Mar 04 '20 at 15:25
  • 1
    this was helpful: https://github.com/edenhill/kafkacat/issues/131 – toto Mar 04 '20 at 15:36