3

I have a Spring Boot application using ReactiveKafkaConsumerTemplate for consuming messages from Kafka.

I've consume messages using kafkaConsumerTemplate.receive() therefore I'm manually acknowledging each message. Since I'm working in an asynchronous manner, messages are not processed sequentially.

I'm wondering how does the commit and poll process work in this scenario - If I polled 100 messages but acknowledged only 99 of them (message not acknowledged is in the middle of the 100 messages I polled, say number 50), what happens on the next poll operation? Will it actually poll only after all 100 messages are acknowledged (and offset is committed) and until then I'll keep getting the un-acknowledged messages over and over to my app until I acknowledge it?

Eyal Ringort
  • 601
  • 6
  • 19

1 Answers1

2

Kafka maintains 2 offsets for a consumer group/partition - the current position() and the committed offset. When a consumer starts, the position is set to the last committed offset.

Position is updated after each poll, so the next poll will never return the same record, regardless of whether it has been committed (unless a seek is performed).

However, with reactor, you must ensure that commits are performed in the right order, since records are not acknowledged individually, just the committed offset is retained.

If you commit out of order and restart your app, you may get some processed messages redelivered.

We recently added support in the framework for out-of-order commits.

https://projectreactor.io/docs/kafka/release/reference/#_out_of_order_commits

The current version is 1.3.11, including this feature.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • So, If I use `basicReceiverOptions.maxDeferredCommits(100).subscription(topicsProvider.getTopics());` I can manually call `record.receiverOffset().acknowledge()` after finished processing a message to commit that message even if it's out-of-order and Kafka will perform the actual commit only when there's no gap, up to a limit of 100 messages (across all the assigned topics/partitions)? – Eyal Ringort Mar 17 '22 at 11:34
  • If so, what happenes if I don't use the new `maxDeferredCommits`? Once there's any gap it will stop polling for messages altogether? – Eyal Ringort Mar 17 '22 at 11:40
  • 1
    It's not Kafka that does it; it is reactor-kafka. Before the feature (or if you don't enable it), it is your responsibility to acknowledge records in the right order. Acknowledging out of order will likely cause lost and/or duplicate records after a failure. With the feature enabled, out of order commits are deferred until gaps are filled. In your case if the deferred commits reaches 100, the consumer is paused (preventing new records arriving) until enough gaps are filled to reduce the deferred commits to less than 100. – Gary Russell Mar 17 '22 at 12:20