Consume the same message again if processing of the message fails

Question

I am using Confluent.Kafka .NET client version 1.3.0. I am following the docs:

var consumerConfig = new ConsumerConfig
{
    BootstrapServers = "server1, server2",
    AutoOffsetReset = AutoOffsetReset.Earliest,
    EnableAutoCommit = true,
    EnableAutoOffsetStore = false,
    GroupId = this.groupId,
    SecurityProtocol = SecurityProtocol.SaslPlaintext,
    SaslMechanism = SaslMechanism.Plain,
    SaslUsername = this.kafkaUsername,
    SaslPassword = this.kafkaPassword,
};

using (var consumer = new ConsumerBuilder<Ignore, string>(consumerConfig).Build())
{
    var cancellationToken = new CancellationTokenSource();
    Console.CancelKeyPress += (_, e) =>
    {
        e.Cancel = true;
        cancellationToken.Cancel();
    };

    consumer.Subscribe("my-topic");
    while (true)
    {
        try
        {
            var consumerResult = consumer.Consume();
            // process message
            consumer.StoreOffset(consumerResult);
        }
        catch (ConsumeException e)
        {
            // log
        }
        catch (KafkaException e)
        {
            // log
        }
        catch (OperationCanceledException e)
        {
            // log
        }
    }
}

The problem is that even if I comment out the line consumer.StoreOffset(consumerResult);, I keep getting the next unconsumed message the next time I Consume, i.e. the offset keeps increasing which doesn't seem to be what the documentation claims it does, i.e. at least one delivery.

Even if I set EnableAutoCommit = false and remove 'EnableAutoOffsetStore = false' from the config, and replace consumer.StoreOffset(consumerResult) with consumer.Commit(), I still see the same behavior, i.e. even if I comment out the Commit, I still keep getting the next unconsumed messages.

I feel like I am missing something fundamental here, but can't figure what. Any help is appreciated!

The messages have already been returned to application from kafka standpoint so when you commit they are saved as last committed offsets but consume will continue to return next messages whether you have consumed or not. What is your expectation here ? Could you please elaborate what you are expecting to happen before/after commit and consume ? — s7vr, Mar 12 '20 at 11:04
Messages are not refetched until you use seek to offset. This will effect the consume and messages will be returned from seek offset. — s7vr, Mar 12 '20 at 11:29
@user2683814 In my post I mentioned two scenarios depending on what `EnableAutoCommit` is set to. Let's say we have `EnableAutoCommit = false`, and when I `Consume`, I get back the message with offset 11. I was expecting to keep getting the same message with offset 11 over and over again if processing the message keeps throwing and hence not call to `Commit` is made. — havij, Mar 12 '20 at 12:10
No,that is not the case. You can't control what to poll (`Consume`) by using `Commit` after you have already `Subscribe` to topic.. Kafka ( as in client lib ) behind the scene maintains all the offsets it has send to app in the `Consume` and it will send them linearly. So to reprocess a message like in a failure scenario you have to track them in your code and seek to offset and starts processing the message and and you should also know what to skip if it has already been processed in earlier requests. I'm not familiar with .net library but it shouldn't really matter as this is kafka design. — s7vr, Mar 12 '20 at 12:23
I think you have to use combination of subscribe and assign and may need different consumers to support your use case. In case of failures use assign/seek to offset for topic partitions with one consumer to reprocess messages and for normal processing use another consumer with subscribe/Consume/Commit flow. — s7vr, Mar 12 '20 at 13:05
Try killing your client mid-way through. Upon restarting you should get the non-committed messages for reprocessing. — Ant, Dec 05 '21 at 02:11

score 1 · Answer 1 · answered Mar 16 '20 at 14:46

You may want to have a re-try logic for processing each of your messages for a fixed number of times like say 5. If it doesn't succeed during these 5 retries, you may want to add this message to another topic for handling all failed messages which take precedence over your actual topic. Or you may want to add the failed message to the same topic so that it will be picked up later once all those other messages are consumed.

If the processing of any message is successful within those 5 retries, you can skip to the next message in the queue.

score 0 · Answer 2 · answered Mar 09 '20 at 04:35

Sorry I can't add comment yet. Kafka consumer consumes message in batchs, so maybe you still iterate through the batch pre-fetched by background thread.

You can check whether your consumer really commit offset or not using kafka util kafka-consumer-groups.sh

kafka-consumer-groups.sh --bootstrap-server kafka-host:9092 --group consumer_group  --describe

SeReGa · Answer 3 · 2021-09-16T16:59:42.057

I had the same situation, and here is my solution:

Set a configuration of max retries for each operation.

For consuming, just retry.
For Saving, re-assign the current offset, and then retry.

Here is the code:

var saveRetries = 0;
var consumeRetries = 0;
ConsumeResult<string, string> consumeResult;

while (true)
{
    try
    {
        consumeResult = consumer.Consume();
        consumeRetries = 0;
    }
    catch (ConsumeException e)
    {
        //Log and retry to consume, up to {MaxConsumeRetries} times
        if (consumeRetries++ >= MaxConsumeRetries)
        {
            throw new OperationCanceledException($"Too many consume retries ({MaxConsumeRetries}). Please check configuration and run the service agian.");
        }
        continue;
    }
    catch (OperationCanceledException oe)
    {
        //Log
        consumer.Close();
        break;
    }

    try
    {
        SaveResult(consumeResult);
        saveRetries = 0;
    }
    catch (ArgumentException ae)
    {
        //Log and retry to save, up to {MaxSaveRetries} times
        if (saveRetries++ < MaxSaveRetries)
        {
            //Assign the same offset, and try again.
            consumer.Assign(consumeResult.TopicPartitionOffset);
            continue;
        }
    }

    try
    {
        consumer.StoreOffset(consumeResult);
    }
    catch (KafkaException ke)
    {
        //Log and let it continue
    }
}

Consume the same message again if processing of the message fails

3 Answers3