1

I want to know whether kafkaTemplate.send(topic, key, message) method will call provided custom partitioner partition() method or not?

amitwdh
  • 661
  • 2
  • 9
  • 19

1 Answers1

1

Well, KafkaTemplate is fully based on the Apache Kafka Client Producer. The final code looks like:

producer.send(producerRecord, buildCallback(producerRecord, producer, future));

The key and topic is really a part of that ProducerRecord. Everything else is done in the KafkaProducer and other Kafka Client objects. In particular the Partitioner.partition() is called from here in the KafkaProducer:

 /**
     * computes partition for given record.
     * if the record has partition returns the value otherwise
     * calls configured partitioner class to compute the partition.
     */
    private int partition(ProducerRecord<K, V> record, byte[] serializedKey, byte[] serializedValue, Cluster cluster) {
        Integer partition = record.partition();
        return partition != null ?
                partition :
                partitioner.partition(
                        record.topic(), record.key(), serializedKey, record.value(), serializedValue, cluster);
    }

And this one is a part of private Future<RecordMetadata> doSend(ProducerRecord<K, V> record, Callback callback) { which is called from the mentioned KafkaTemplate.doSend().

You may consider to use this API instead:

/**
 * Send the data to the provided topic with the provided key and partition.
 * @param topic the topic.
 * @param partition the partition.
 * @param timestamp the timestamp of the record.
 * @param key the key.
 * @param data the data.
 * @return a Future for the {@link SendResult}.
 * @since 1.3
 */
ListenableFuture<SendResult<K, V>> send(String topic, Integer partition, Long timestamp, K key, V data);

So, you have the full control in your code over partition as well.

But shortly: KafkaTemplate doesn't call Partitioner.partition().

On the other hand: why just don't try that in your project? you might even haven't come to us with such a question...

Artem Bilan
  • 113,505
  • 11
  • 91
  • 118
  • The final partition depends on the partitioner. The default one is a no-op if the record already has one. So it is up to you what you do in your custom one. – Gary Russell Sep 04 '19 at 13:47
  • @Artem Bilan: So KafkaTemplate doesn't call partition() method even if we give Key in send method? Basically I have written custom Partitioner by implementing Partitioner Interface. I want partition() method to get call when I will provide key in send method. – amitwdh Sep 05 '19 at 14:19
  • Well, OK. It is called from the `KafkaTemplate.doSend()` call stack. It is not directly from there, but eventually it is called in `KafkaProducer`. Still not clear why you haven't tried that on your side before coming to us with the question... – Artem Bilan Sep 05 '19 at 14:21
  • Thank you @Artem. Spending more efforts on task with some other reasons, so thought of posting here to same efforts. – amitwdh Sep 05 '19 at 15:22