I have a recently upgraded Kafka Streams 2.4.1 application and we have started seeing the application periodically crash due to the following error:
2020-05-13T11:27:08.553Z INFO <> [chat-98346af0-1be5-41e9-b15f-3cba364efa35-StreamThread-2] o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=chat-98346af0-1be5-41e9-b15f-3cba364efa35-StreamThread-2-consumer, groupId=chat] Found no committed offset for partition private.chat.endpoint-1
2020-05-13T11:27:08.554Z ERROR <> [chat-98346af0-1be5-41e9-b15f-3cba364efa35-StreamThread-2] o.a.k.s.p.internals.StreamThread - stream-thread [chat-98346af0-1be5-41e9-b15f-3cba364efa35-StreamThread-2] Encountered the following error during processing:
java.lang.IllegalStateException: Offset limit should monotonically increase, but was reduced. New limit: 0. Previous limit: 1641
at org.apache.kafka.streams.processor.internals.StandbyTask.updateOffsetLimits(StandbyTask.java:215)
at org.apache.kafka.streams.processor.internals.StandbyTask.update(StandbyTask.java:181)
at org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:1009)
at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:825)
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:698)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:671)
What can cause 'Found no committed offset for partition'?
Other things to note:
- We have TOPOLOGY_OPTIMIZATION set to OPTIMIZE
- We are only seeing this on two of our three environments (at the moment)
- There is some evidence to suggest there is a pattern to the timing of this error. Although not always the case, 24hrs between errors is common.
