0

Try to follow the instruction on internet to achieve kafka asynchronous produce. Here is what my producer looks like:

import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerRecord;
public void asynSend(String topic, Integer partition, String message) {
    ProducerRecord<Object, Object> data = new ProducerRecord<>(topic, partition,null, message);
        producer.send(data, new DefaultProducerCallback());
    }

private static class DefaultProducerCallback implements Callback {
    @Override
    public void onCompletion(RecordMetadata recordMetadata, Exception e) {
        if (e != null) {
            logger.error("Asynchronous produce failed");
        }
    }
}

And I produce in a for loop like this:

for (int i = 0; i < 5000; i++) {
    int partition = i % 2;
    FsProducerFactory.getInstance().asynSend(topic, partition,i + "th message to partition " + partition);
}

However, some message may get lost. As shown below, message from 4508 to 4999 not delivered.

enter image description here

I find the reason might be the shutdown of producer process and all message in cache not send at that time would be lost. Add this line after for loop would solve this problem:

producer.flush();

However, I am not sure whether it is a charm solution because I notice someone mentioned that flush would make Asynchronous send somehow Synchronous, can anyone explain or help me improve it?

Michael Heil
  • 16,250
  • 3
  • 42
  • 77
Fallin
  • 45
  • 10

1 Answers1

0

In the book Kafka - The definitive Guide there is an example for an asznchronous Producer given exactly as you have written the code. It uses send together with a Callback.

In a discussion it is written:

Adding flush() before exiting will make the client wait for any outstanding messages to be delivered to the broker (and this will be around queue.buffering.max.ms, plus latency). If you add flush() after each produce() call you are effectively implementing a sync producer.

But if you do it after the for loop it is not synchronous anymore but rather asynchronous.

What you could do also do is to set the acks in the Producer configuration to all. That way you will have some more guarantees to successfully produce messages in case the replication of the topic is set to greater than 1.

Michael Heil
  • 16,250
  • 3
  • 42
  • 77
  • thx mike, `flush()` now seems not a bad choice to keep producer alive and get all message delivered in my scenario. I also tried `acks` which you suggest but did not work. – Fallin Apr 21 '20 at 01:54
  • Have you checked all partitions to see if all data was produced? In your screenshot it only shows partition 1 but nothing about partition 0. Keep in mind that the order could be different over partitions. – Michael Heil Apr 21 '20 at 04:21
  • my consumer thread polls every 100ms and consume about 800 messages each poll. So there is 800 messages in a row in the output for a specific partition, followed by 800 messages for another partition. I am sure that data of both partition was produced, but none of each was fully delivered. – Fallin Apr 21 '20 at 06:50