Custom Partitioner for Plain Producer | Kafka Streams

Question

I have an kafka streams application which has

props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, MyPartitioner.class);

or

props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, RoundRobinPartitioner.class);

which is a class for distributing messages to different partitions even using same key in kafka 2.4 version

RoundRobinPartitioner has this implementation:

public class RoundRobinPartitioner implements Partitioner {
    private final ConcurrentMap<String, AtomicInteger> topicCounterMap = new ConcurrentHashMap();

    public RoundRobinPartitioner() {
    }

    public void configure(Map<String, ?> configs) {
    }

    public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
        List<PartitionInfo> partitions = cluster.partitionsForTopic(topic);
        int numPartitions = partitions.size();
        int nextValue = this.nextValue(topic);
        List<PartitionInfo> availablePartitions = cluster.availablePartitionsForTopic(topic);
        if (!availablePartitions.isEmpty()) {
            int part = Utils.toPositive(nextValue) % availablePartitions.size();
            return ((PartitionInfo)availablePartitions.get(part)).partition();
        } else {
            return Utils.toPositive(nextValue) % numPartitions;
        }
    }

    private int nextValue(String topic) {
        AtomicInteger counter = (AtomicInteger)this.topicCounterMap.computeIfAbsent(topic, (k) -> {
            return new AtomicInteger(0);
        });
        return counter.getAndIncrement();
    }

    public void close() {
    }
}

and my Partitioner consists of exactly same code but different partition method implementation and my code block is:

    public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
        List<PartitionInfo> partitions = cluster.partitionsForTopic(topic);

        int numPartitions = partitions.size();

        int nextValue = nextValue(topic);

        return Utils.toPositive(nextValue) % numPartitions;

    }

When I configure like that messages are distributed to different partitions, in both implementation, but never use the some partitions.

I have 50 partitions and partition 14 and 34 never received a message. My partitions are not unavaliable. They are avaliable. When I change my return partition method to 14 or 34, all my messages go to that partition. What could be the problem ? Both implementations are not working as expected.

Edit 1: I have tried RoundRobinPartitioner with the plain producer. Result is the same. Producer can not produce messages equally amoung partitions, some partitions are never used. What could be the reason ? It is not like a missing configuration.

Edit 2: I have debug RoundRobinPartitioner and put a breakpoint at return. When I produce just 1 message, Producer produce message twice. First attempt is always unsuccessful and that message does not go any partition. When I hit continue at debugging index of the ConcurrentMap increases by 1. The second attempt of the producer is successful.

partition() method is invoked something where I could not find yet.

Edit 3: Could this be related with onNewBatch method which I did not override ?

Edit 4: This implementatin works for kafka client 2.2 but not for 2.4. Partition interface does not have onNewBatch method. DefaultPartitioner implementation is changed when key is null 2.2 vs 2.4. Can it be related with stick partitions ?

Nope, they all have same key, but does it matter ? because I am simply returning getAndIncrement with AtomicInteger so the message go to that partition. My class extends Partitioner class — Alpcan Yıldız, Feb 03 '20 at 14:49
When using the built-in `RoundRobinPartitioner` and sending messages with the _same_ key; are they placed in different partitions? — Giorgos Myrianthous, Feb 03 '20 at 14:58
Nope, They are placed in the same partitions, nothing is changed — Alpcan Yıldız, Feb 03 '20 at 15:00
Let's say I have a same key for 10.000 records, for 10.000 another key.. Not using exactly the same key but I have same key for batch of records. These records can be change to millions. I have triggered my records 4-5 different times for above image. That means I have used ~5 or 6 different key. — Alpcan Yıldız, Feb 03 '20 at 15:06
Does this answer your question? [Kafka Streams RoundRobinPartitioner streaming config](https://stackoverflow.com/questions/60004267/kafka-streams-roundrobinpartitioner-streaming-config) — OneCricketeer, Feb 06 '20 at 13:40
Nope. I have sent my messages without **key** with RoundRobinPartitioner. It is still not working :/ — Alpcan Yıldız, Feb 07 '20 at 07:58
Maybe you are hitting https://issues.apache.org/jira/browse/KAFKA-9965 ? — Matthias J. Sax, Jul 08 '20 at 22:06

score 3 · Accepted Answer · answered Feb 21 '20 at 10:17

3

Use UniformStickyPartitioner.class in kafka 2.4 client version. RoundRobinPartitioner.class works for kafka 2.2 or lower versions. In 2.4 version

props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, UniformStickyPartitioner.class);

should be used. I think this is related with new StickPartitioner.

answered Feb 21 '20 at 10:17

Alpcan Yıldız

671
3
13
33

2

Maybe you are hitting https://issues.apache.org/jira/browse/KAFKA-9965 ? – Matthias J. Sax Jul 08 '20 at 22:06
2

Yes exactly that was my issue ! When I wrote this issue, it was february :) I am marking as an answer, thanks Matthias – Alpcan Yıldız Jul 08 '20 at 22:34
2

I spent 2 days on this problem going crazy, thank you! :) – Cemre Mengü Oct 31 '21 at 11:48

Custom Partitioner for Plain Producer | Kafka Streams

1 Answers1

Linked