After all Kafka brokers restart to upgrade offset.retention.minutes setting (to increase it to 60 days), the Kafka Streams application consuming there were stuck, and the consumer group shows rebalancing:
bin/kafka-consumer-groups.sh --bootstrap-server ${BOOTSTRAP_SERVERS} --describe --group stream-processor | sort
Warning: Consumer group 'stream-processor' is rebalancing.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
customers 0 84263 84288 25 - - -
customers 1 71731 85068 13337 - - -
customers 10 71841 84801 12960 - - -
customers 11 84273 84336 63 - - -
customers 12 84189 84297 108 - - -
customers 13 83969 84051 82 - - -
customers 14 84693 84767 74 - - -
customers 15 84472 84556 84 - - -
customers 2 84175 84239 64 - - -
customers 3 71719 71719 0 - - -
customers 4 71446 84499 13053 - - -
customers 5 84291 84361 70 - - -
customers 6 71700 71700 0 - - -
customers 7 72003 85235 13232 - - -
customers 8 84521 84587 66 - - -
customers 9 71513 71513 0 - - -
customers-intermediate 0 102774 102792 18 - - -
customers-intermediate 1 102931 103028 97 - - -
customers-intermediate 10 102883 102965 82 - - -
customers-intermediate 11 102723 102861 138 - - -
customers-intermediate 12 102797 102931 134 - - -
customers-intermediate 13 102339 102460 121 - - -
customers-intermediate 14 103227 103321 94 - - -
customers-intermediate 15 103231 103366 135 - - -
customers-intermediate 2 102637 102723 86 - - -
customers-intermediate 3 84620 103297 18677 - - -
customers-intermediate 4 102596 102687 91 - - -
customers-intermediate 5 102980 103071 91 - - -
customers-intermediate 6 84385 103058 18673 - - -
customers-intermediate 7 103559 103652 93 - - -
customers-intermediate 8 103243 103312 69 - - -
customers-intermediate 9 84211 102772 18561 - - -
events 15 11958555 15231834 3273279 - - -
events 3 1393386 16534651 15141265 - - -
events 4 1149540 15390069 14240529 - - -
visitors 15 2774874 2778873 3999 - - -
visitors 3 603242 603324 82 - - -
visitors 4 565266 565834 568
The streaming application was restarted too, and afterwards I could see some processing logs for about 20 hours and then stopped processing.
It's been like this for two days. But it is also worth mentioning that all topics you see above have 16 partitions, but some show three of them (visitors, events). However I can describe the topics and they have their partitions well distributed as usual and I can find nothing strange there.
What could have happened?
After application restart, I can see all partitions again, and the applications consuming from topic partitions. However many (most) partitions had lost their offsets. Since I changed the offset.retention.minutes setting, this should not have happened.
events 0 - 14538261 - stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer-a8a6e989-d6c1-472f-aec5-3ae637d87b9e /ip2 stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer
events 1 49070 13276094 13227024 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 10 - 15593746 - stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 11 - 15525487 - stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 12 - 21863908 - stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 13 - 15810925 - stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 14 - 13509742 - stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 15 11958555 15231834 3273279 stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer-a8a6e989-d6c1-472f-aec5-3ae637d87b9e /ip2 stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer
...
Kafka 1.1.0, Kafka Streams 1.0
UPDATE Still happening on 2.1.1.
