0

I created a kafka consumer that executes on a batch process for a single partitioned topic. Every 15 minutes, my batch process will execute and consume all new messages that have been published to the topic. After consuming all available messages at that time, the batch process will exit.

Below is sample code to how I achieved this behavior:

CancellationTokenSource cts = new CancellationTokenSource();
bool isPartitionEof = false;

using (var consumer = new ConsumerBuilder<Ignore, string>(consumerConfig).Build())
{
    consumer.Subscribe("topicName");
    while (!isPartitionEof)
    {
        ConsumeResult<Ignore, string> consumeResult = consumer.Consume(cts);
        isPartitionEof = consumeResult.IsPartitionEof;

        if (consumeResult.Message != null)
        {
          // consume logic
        }
    }
}

Once the topic goes to multi-partitions the logic above will no longer work due to hitting the end of one partition will pre-maturely exit the batch application. Is there a way that I can iterate through the partitions on a topic and consume messages on a partition basis?

Most examples I find online are service-oriented Kafka consumers that infinitely poll and consume when a message is published to the topic. Unfortunately, my situation is a bit more unique, which requires me to have an explicit condition to stop reading new messages once all have been consumed and exit the batch application. Any help would be greatly appreciated!

Evan Zimmerman
  • 1
  • Do you have the ability to create the same number of consumer threads as there are partitions? Each consumer thread will have its own partition to read from then. – franzke Apr 12 '21 at 20:33
  • No I do not control the Kafka cluster what so ever. I did end up finding a solution by simply passing a TimeSpan of 10 seconds to the consume method. This would then do one final check to all partitions to see if there is another message available. If not, it simply kicked out of the infinite while loop. This solution is not ideal if you have a high throughput topic that publishes messages before you have the chance to "timeout". – Evan Zimmerman May 06 '21 at 02:12

0 Answers0