0

I've seen an issue where all my messages in my topic gets re-read by my consumer. I only have 1 consumer, and I turn it on/off while I'm developing/testing. I notice that sometimes after days of not running the consumer, when I turn it on again suddenly it re-reads all my messages.

The clientid and groupid stays the same throughout. I explicitly call commitSync, since my enable.auto.commit=false. I do set auto.offset.reset=earliest, but to my understanding that should only kick in if the offset is deleted on the server. I'm using IBM Bluemix's MessageHub service, so maybe that's automatically deleting an offset?

Does anyone have any clues/ideas?

Thanks.

kyl
  • 475
  • 1
  • 5
  • 16

1 Answers1

1

Yes offsets are automatically deleted if you don't commit for 24hours. This is the default setting with Kafka and we've not changed it.

Mickael Maison
  • 25,067
  • 7
  • 71
  • 68
  • Oh I see, I've found the setting too, offsets.retention.minutes. Is this configureable in bluemix for the broker? Thanks a lot for your help! – kyl Jun 07 '17 at 20:17
  • At the moment, it's not configurable. Although the Kafka protocol allows to specify a retention time when commiting offsets, this fonctionality is not exposed in the clients. – Mickael Maison Jun 08 '17 at 12:44