There is a weird issue I have fallen into where I have created a kafka consumer in dotNet and have disabled the auto commit
config = new Confluent.Kafka.ConsumerConfig
{
BootstrapServers = "localhost:9092",
GroupId = "Group1",
AutoOffsetReset = AutoOffsetReset.Earliest,
EnableAutoCommit = false,
AutoCommitIntervalMs=0,
EnableAutoOffsetStore = false,
};
The idea here is to consume records in sequence and if the processing of a kafka record fails then do not commit that message and re-read the same message.
while(True){
//Consume the message -> c is my consumer
cr = c.Consume(10);
if(processCR(cr)){
//do something
c.Commit(cr)
}
else{
//do something
// do not commit the message
}
}
Input
0
1
2
3
4
5
6
7
now suppose the record 5
fails
then the expected consumption is
0
1
2
3
4
5
5 // re-consume and process successfully
6
7
But the sequence is which I am actually getting is
0
1
2
3
4
5
6
7
This is something weird and have been stuck in a loop tried out multiple combination of config but nothing works.
Any lead is appreciated !!