5

Can I implement manual Kafka offset management with Spring Cloud Steam as follows:

  1. Whenever my consumer processes a message, it commits its offset into DB. Not into Kafka
  2. Whenever my consumers restarts, it reads last processed offset from the DB, seeks to that offset and starts processing next messages.
Nikem
  • 5,716
  • 3
  • 32
  • 59

1 Answers1

3

Spring Cloud Stream lets you manually acknowledge the messages at the consumer application. Not sure why you want to persist the offset into DB (may be that's your need). But I am wondering if using manual ack mode helps your case.

You can use manual ack mode by setting spring.cloud.stream.kafka.bindings.<inboundChannelName>.consumer.autoCommitOffset to false and manually acknowledge the messages only after the consumer processed the messages. You can find an example here

Ilayaperumal Gopinathan
  • 4,099
  • 1
  • 13
  • 12
  • 2
    The problem lies in restarts. I want to strictly control where I start after consumer is restarted, in order to have exactly-once delivery. – Nikem Nov 16 '16 at 11:02
  • When combining manual acknowledge mode and starting from the `latest` offset during the restart of the consumer, doesn't that help achieve `exactly-once` delivery? – Ilayaperumal Gopinathan Nov 16 '16 at 11:31
  • The underlying spring-kafka project (when using kafka 10) supports this via the [seek mechanism](http://docs.spring.io/spring-kafka/docs/1.1.1.RELEASE/reference/html/_reference.html#seek) but Spring Cloud Stream does not currently expose it as a feature. – Gary Russell Nov 16 '16 at 13:58
  • There's an open issue for this in the Kafka binder: https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/63 – Marius Bogoevici Nov 16 '16 at 15:07
  • Do note however that resetting offsets and replay would properly work only for a single consumer application (or in Spring Cloud Stream for statically allocated partitions) - or else it ends up in a race condition as multiple instance launch. – Marius Bogoevici Nov 16 '16 at 15:08