0

There is a weird issue I have fallen into where I have created a kafka consumer in dotNet and have disabled the auto commit

config = new Confluent.Kafka.ConsumerConfig
            {
                BootstrapServers = "localhost:9092",
                GroupId = "Group1",
                AutoOffsetReset = AutoOffsetReset.Earliest,
                EnableAutoCommit = false,
                AutoCommitIntervalMs=0,
                EnableAutoOffsetStore = false,  
        };

The idea here is to consume records in sequence and if the processing of a kafka record fails then do not commit that message and re-read the same message.

while(True){
//Consume the message -> c is my consumer

 cr = c.Consume(10);
 if(processCR(cr)){
       //do something
       c.Commit(cr)
 }
 else{
       //do something
       // do not commit the message
 }
}

Input

0
1
2
3
4
5
6
7

now suppose the record 5 fails then the expected consumption is

0
1
2
3
4
5
5 // re-consume and process successfully
6
7

But the sequence is which I am actually getting is

0
1 
2
3
4
5
6
7

This is something weird and have been stuck in a loop tried out multiple combination of config but nothing works.

Any lead is appreciated !!

Bimal Gangawal
  • 385
  • 2
  • 17

0 Answers0