Spring Cloud Stream Kafka Producer force producer.flush

Question

I am creating a kafka streams/in-out kind of application. Sample code looks like the following

private MessageChannel output;

public void process(List<String> input) {
--somelogic
output.send()
}

Based on my understanding, kafka buffers the messages before sending them out. Now in case the app crashes / container crashes, there is a possibility of losing the buffered messages.

How can we ensure that messages are actually sent out

(something like KafkaTemplate.flush() here ?)

EDIT Based on the suggestion by Gary Russell, we should set the FLUSH header on the last message.

Follow up question - Given the last send method call will become a blocking call because of the blocking nature of the kafkaProducer.flush() method, if there are exceptions thrown during the sending process of the kafka message (e.g. io exception/ auth exception) will they be raised in the same method context ?

e.g. in the following code will the kafka sender exception be caught in the catch block ?

public void process(List<String> input) {
--somelogic
  try {
  Message<?> message = //
  message.setHeader(KafkaIntegrationHeaders.FLUSH, true);
  output.send()
 }
catch(Exception e) {
  e.printstacktrace()
}
}

Kafka only buffers output if `linger.ms` is increased from its default 0. — Gary Russell, Jun 06 '22 at 13:40
in general linger.ms would be > 0 to reduce network traffic, here i am trying to ensure that messages are sent before completing the processing for the batch — Rachit Agrawal, Jun 06 '22 at 22:18

score 0 · Accepted Answer · answered Jun 06 '22 at 22:34

0

The underlying KafkaProducerMessageHandler has a property:

/**
 * Specify a SpEL expression that evaluates to a {@link Boolean} to determine whether
 * the producer should be flushed after the send. Defaults to looking for a
 * {@link Boolean} value in a {@link KafkaIntegrationHeaders#FLUSH} header; false if
 * absent.
 * @param flushExpression the {@link Expression}.
 */
public void setFlushExpression(Expression flushExpression) {

Currently, the binder doesn't support customizing this property.

However, if you are sending Message<?>s, you can set the KafkaIntegrationHeaders.FLUSH header to true on the last message in the batch.

answered Jun 06 '22 at 22:34

Gary Russell

166,535
14
146
179

thanks, one more question - so the last send method call becomes a blocking call right ? (because of flush on the last message) – Rachit Agrawal Jun 06 '22 at 22:37
Flush on the producer will block, yes; see the javadocs for `KafkaProducer.flush()` for more information. – Gary Russell Jun 06 '22 at 22:42
Updated the question with follow-up question, can you pls answer that as well – Rachit Agrawal Jun 06 '22 at 22:52
Yes, see the javadocs; flush will wait on all the outstanding futures to complete; if there is a failure, the exception is thrown to the caller. – Gary Russell Jun 07 '22 at 13:03

Spring Cloud Stream Kafka Producer force producer.flush

1 Answers1