Kafka consumer messages best practices

Question

Net client. I have below implementation for consumer.

while (true)
                {
                    try
                    {
                        _logger.LogInformation("started consuming");
                        var consumeResult = consumer.Consume(TimeSpan.FromSeconds(10));

                        if (consumeResult == null)
                        {
                            break;
                        }

                        if (consumeResult.IsPartitionEOF)
                        {
                            _logger.LogInformation(
                                $"Reached end of topic {consumeResult.Topic}, partition {consumeResult.Partition}, offset {consumeResult.Offset}.");
                            break;
                        }

                        string message = JsonConvert.SerializeObject(consumeResult.Message.Value);
                        fixureData = JsonConvert.DeserializeObject<U>(message)!;
                        listMessages.Add(fixureData);
                        _logger.LogInformation("Consumed Message: " + message);
                    }
                    catch (ConsumeException e)
                    {
                        _logger.LogError($"ConsumeException occurred while consuming RETinA messages: {e.Error.Reason}");
                    }
                }

I have this code in console application and running it as job. My idea is running this job once a day to receive all the messages. Is this is right approach or do i need to create background/hosted service which will keep listening to consume? Which would be best approach? Also If i have cancellation token as like below

CancellationToken cancellationToken = new CancellationToken();
 while (true)
                {
                    try
                    {
                        _logger.LogInformation("started consuming");
                        var consumeResult = consumer.Consume(cancellationToken);
                        //rest of the code
                    }
                 }

If I have above implementation, then always my consumer will be trying to consume but once all the messages consumer I would like to end this loop but how to detect all messages consumed? I have three partitions. So how can I make sure all the messages consumed successfully? Can someone please help me here?

Kafka consumer messages best practices

0 Answers0