9

I am using Spring kafka for the first time and I have created a Producer and Consumer by using spring kafka. I have my kafka server running on localhost and have created a topic called test. I was not able to send messages to the consumer by simply calling

        KafkaTemplate.send(topicName,Data);

I had to call flush() on kafkaTemplate after calling send on the same object and then the consumer was able to receive the data. Okay it works and it is fantastic. But could anyone explain to me what is happening behind the scenes? Why is the flush method required to be called.

From the official spring kafka documentation.

           public void flush()

Flush the producer. Note It only makes sense to invoke this method if the ProducerFactory serves up a singleton producer (such as the DefaultKafkaProducerFactory).

Thank you in advance.

Indraneel Bende
  • 3,196
  • 4
  • 24
  • 36
  • Try to set `ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION` to `1` in your producer factory config. – donm Jan 10 '18 at 17:04
  • `kafkaTemplate.send` returns `Future`. You can write `KafkaTemplate.send(topicName,Data).completable().join()` and you will wait for the thread to be completed. I am trying to understand in which case it is better to use `join` or `flush` – dmitryvim Sep 15 '21 at 15:16

1 Answers1

16

the implement of producer is async. Message is stored in an internal queue to wait to send by inner thread, which would improve efficiency with potential batching.

So, messages may stay in client's memory when program exit. In this case, Kafka server don't actually receive these messages.

message would be sent in timeout defined by queue.buffering.max.ms, or other size / number limit.

flush force all message in send queue to be delivered to server.

neuo
  • 653
  • 4
  • 10
  • Thank u for your answer. So if I dont flush and the program exits, when will the message get actually delivered? I know u talked about queue.buffering.max.ms, but shouldnt it have some default value after which it sends the message? – Indraneel Bende Jan 08 '18 at 14:04
  • @IndraneelBende, `queue.buffering.max.ms` default value is 5000ms – neuo Jan 08 '18 at 14:33
  • Yes, just by ensuring the program does not immediately exit after the send method is called on kafkaTemplate( just put a delay), the consumer was able to receive the message. Should have remembered it is asynchronous! Thanks for ur input! – Indraneel Bende Jan 09 '18 at 04:54
  • @IndraneelBende, you're welcome. Please accept this answer if it's useful . – neuo Jan 09 '18 at 05:42