In a recent common library upgrade, I found Kafka consumer doesn't received message within the expected time(2 minutes). After compared with the code and did lots of tests, looks it is caused by auto.offset.reset=latest(it was earliest). What would be the root cause?
Env:
Spring-Boot 2.7.11; spring-kafka-test 2.8.11
main Consumer config:
max.poll.records=250
max.poll.interval.ms=300000
heartbeat.interval.ms=3000
enable.auto.commit=false
auto.offset.reset=latest
auto create topic = true
can't copy the code, but it mainly uses:
ConcurrentKafkaListenerConntainerFactory
@KafkaListener annotation
the main change is auto.offset.reset=earliest to latest
follow-up: refer to Embedded Kafka Test - The Kafka listener is not invoked
EmbeddedKafka sending messages to consumer after delay in subsequent test
gary russell mentioned that it might send the message before the container completely starts. so what will happen in this case?
- does the message arrived at the topic?
- with "offset=latest", it won't point to the message I sent?
update again(May/19):
as suggested, I added these code(but keep auto.offset.reset=latest):
ContainerTestUtils.waitForAssignment(container, 1) // configured 1 partition in embeddedkafka
However, I saw in log:
Publishing records xxx here
o.a.k.c.c.internals.SubscriptionState: ....Resetting offset for partition xxxx to position FetchPosition(offset=2,...)
I supposed offset reset should happen before publishing and offset=0?
