1

I have created a topic first_topic and produced messages to it.

for (int i = 0; i < 10; i++) {
    //create producer record
    ProducerRecord<String, String> record =
            new ProducerRecord<String, String>("first_topic", "hello world " + i);
    //send Data
    producer.send(record, new Callback() {
        public void onCompletion(RecordMetadata recordMetadata, Exception e) {
            //executes every time a record is send or an exception occurs
            if (e == null) {
                //the record was successfully sent
                logger.info("Received new meta data \n" +
                        "Topic : " + recordMetadata.topic() + "\n" +
                        "Partition : " + recordMetadata.partition() + "\n" +
                        "OFfset : " + recordMetadata.offset() + "\n" +
                        "Timestamp : " + recordMetadata.timestamp());
            } else {
                e.printStackTrace();
                logger.error("Error while Producing record ", e);
            }
        }
    });
}

But all messages go to the partition #2. Ideally they should go to all 3 in round robin way. But no. See below. What am I doing wrong?

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-third-application

Consumer group 'my-third-application' has no active members.

GROUP                TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
my-third-application first_topic     0          0               0               0               -               -               -
my-third-application first_topic     1          0               0               0               -               -               -
my-third-application first_topic     2          10              10              0               -               -               -
Learner
  • 65
  • 4
  • Have you tried sending more than just 10 records? The producer batches together records in one request. – OneCricketeer Dec 28 '19 at 06:43
  • 1
    Yes i think that is the rational that it is batching all the records from one request (all 10 in loop) to one partition. If I run it again it will go to other but all together. So that seems right. Thx. – Learner Dec 29 '19 at 15:39
  • 1
    I had a similar issue. If you would like to omit the batching and force to send every message separately, you can place the producer.flush() method call inside the for loop. Then it should be evenly distributed across partitions. – Marcin Borowski Nov 21 '20 at 14:34

1 Answers1

0

I faced this problem too. the cause is not the round-robin partitioner but the producer "doSend" method causes this problem. in "doSend" method when the accumulator returns the result with abortForNewBatch flag with true value the doSend method calls "partition" method again and the previously selected partition remains unused. this problem is dangerous if the topic had only two partitions because in this case, only one partition will be used.

doSend Method:

...

    RecordAccumulator.RecordAppendResult result = accumulator.append(tp, timestamp, serializedKey,
                        serializedValue, headers, interceptCallback, remainingWaitMs, true, nowMs);
    
                if (result.abortForNewBatch) {
                    int prevPartition = partition;
                    partitioner.onNewBatch(record.topic(), cluster, prevPartition);
                    partition = partition(record, serializedKey, serializedValue, cluster);
                    tp = new TopicPartition(record.topic(), partition);
                    if (log.isTraceEnabled()) {
                        log.trace("Retrying append due to new batch creation for topic {} partition {}. The old partition was {}", record.topic(), partition, prevPartition);
                    }
                    // producer callback will make sure to call both 'callback' and interceptor callback
                    interceptCallback = new InterceptorCallback<>(callback, this.interceptors, tp);
    
                    result = accumulator.append(tp, timestamp, serializedKey,
                        serializedValue, headers, interceptCallback, remainingWaitMs, false, nowMs);
                }
...

this problem will correct with using a custom round-robin partitioner like this:

public class CustomRoundRobinPartitioner implements Partitioner {

    private final ConcurrentMap<String, AtomicInteger> topicCounterMap = new ConcurrentHashMap<>();
private final ConcurrentMap<String, AtomicInteger> unusedPartition = new ConcurrentHashMap<>();

@Override
public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
    if (unusedPartition.containsKey(topic))
        return unusedPartition.remove(topic).get();

    return nextPartition(topic, cluster);
}

public int nextPartition(String topic, Cluster cluster) {
    List<PartitionInfo> partitions = cluster.partitionsForTopic(topic);
    int numPartitions = partitions.size();
    int nextValue = counterNextValue(topic);
    List<PartitionInfo> availablePartitions = cluster.availablePartitionsForTopic(topic);
    if (!availablePartitions.isEmpty()) {
        int part = Utils.toPositive(nextValue) % availablePartitions.size();
        return availablePartitions.get(part).partition();
    } else {
        // no partitions are available, give a non-available partition
        return Utils.toPositive(nextValue) % numPartitions;
    }
}

private int counterNextValue(String topic) {
    AtomicInteger counter = topicCounterMap.computeIfAbsent(topic, k -> {
        return new AtomicInteger(0);
    });
    return counter.getAndIncrement();
}

@Override
public void close() {
}

@Override
public void configure(Map<String, ?> configs) {
}

@Override
public void onNewBatch(String topic, Cluster cluster, int prevPartition) {
    unusedPartition.put(topic, new AtomicInteger(prevPartition));
}
}
yousef
  • 92
  • 1
  • 10