20

I have created a topic that has many partitions. Using the console producer I want to send messages to particular partitions and view the through the console consumer. On the console producer I have tried this,

kafka-console-producer.bat --broker-list localhost:9092 --topic sample  --property parse.key=true --property key.separator=,

Send messages as,

key1,another-message

But I am just confused on whether key1 represents partition number.

Using the console consumer I viewed the messages,

kafka-console-consumer.bat --zookeeper localhost:2181 --topic sample

I want to view the messages according to the partitions. Is this the right way to view the messages on the console consumer? Can anyone please provide a clear understanding on this?

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
Amrutha Jaya Raj
  • 612
  • 2
  • 10
  • 32
  • 1
    Note that if you _are not_ using the console producer but a Kafka producer client (like the official Java producer client) you can actually directly specify the target partition that a message should be sent to. The console producer is nice for playing around but typically not used in production. – miguno May 14 '18 at 10:13
  • Possible duplicate of [How to produce messages to selected partition using kafka-console-producer?](https://stackoverflow.com/questions/26553412/how-to-produce-messages-to-selected-partition-using-kafka-console-producer) – Jacek Laskowski Jul 17 '19 at 15:33

1 Answers1

26

You can specify partition number directly in the ProducerRecord, but not with kafka-console-producer.

The key is not the partition number but Kafka uses the key to specify the target partition. The default strategy is to choose a partition based on a hash of the key or use round-robin algorithm if the key is null.

If you need a custom algorithm to map the messages to partitions, you need to implement org.apache.kafka.clients.producer.Partitioner interface. The name of you class must be set as a partitioner.class property of the producer.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Katya Gorshkova
  • 1,483
  • 9
  • 16
  • 4
    Custom partitioning can be done with programming, one of the examples can be found [here](https://howtoprogram.xyz/2016/06/04/write-apache-kafka-custom-partitioner/). You cannot specify it with a kafka-console-producer – Katya Gorshkova May 14 '18 at 10:58
  • But this example is a Java code, do you have the same in Scala? – Amrutha Jaya Raj May 15 '18 at 07:44
  • Is there any way so that we can specify partition number while sending? – Amrutha Jaya Raj May 21 '18 at 06:53
  • 3
    Yes, create a ProducerRecord with `ProducerRecord(java.lang.String topic, java.lang.Integer partition, K key, V value) Creates a record to be sent to a specified topic and partition` – Katya Gorshkova May 21 '18 at 08:08
  • So how can we identify the partition number? can you give an example? – Amrutha Jaya Raj May 21 '18 at 10:04
  • What is the difference in using partition number and key. I am getting confused. – Amrutha Jaya Raj May 21 '18 at 11:12
  • 2
    Producer uses the hash of the key to distribute the message to partitions. For example, you can have two partitions, the integer key and have algorithm to distribute the even keys to the first partition and the odd keys to the second partition. As for partition number, when you send the message to Kafka you use ProducerRecord object. With this object you can specify partition number explicitly. – Katya Gorshkova May 21 '18 at 12:15
  • Ok I got it. Thank you – Amrutha Jaya Raj May 22 '18 at 05:22
  • Please could you advise what happens when the key (or hash) is the same for more than one messages? That is, let's say I have 3 partitions, and I try to write two messages with the exact same key to a topic? Does it matter if the message body is the same or different? Thanks! – Dhruv Saxena Nov 24 '20 at 19:36
  • 4
    Keys in Kafka are not unique. So, if you have 3 messages with the same key, all the 3 messages will be written into Kafka. It does not matter if they have the same or different body. If you have partitioning algorithm based on key, all the 3 messages will go to the same partition. – Katya Gorshkova Nov 25 '20 at 08:58