what will happen after the Kafka consumer reads the whole partion?

Question

I do have a .Net code that I'm using in a windows service that has the process of initializing a consumer object and consuming the Kafka.

var config = new ProducerConfig
{
    BootstrapServers = "host1:9092,host2:9092,...",
    ClientId = ...,
    ...
};


ConsumerBuilder<byte[], byte[]> c = new ConsumerBuilder<byte[], byte[]>(config ).SetErrorHandler(_Consumer_OnError);

consumer = c.Build();

consumer.Subscribe(topics);

while (!canceled)
{
      var consumeResult = consumer.Consume(cancellationToken);
      //handle consumed message.
      ...
}

I want to know that what will happen after all the messages are consumed from the partition. FYI: There are 3 partitions and 4 nodes for consuming the information.

score 0 · Answer 1 · answered Jan 23 '21 at 14:57

From confluent-kafka-dotnet, ConsumeResult<T1, T2> Consume(CancellationToken ct) method will:

Poll for new messages/events. Blocks until a consume result is available or the operation has been cancelled.

This means calling Consume() will never really stop trying to consume messages from the partition (unless something crashes or you cancel the op). So if you have two events in a partition, the third call on Consume() will just block until a new message is written to the partition.

Under the hood during the Consume method, Kafka client keeps making calls to the broker to check whether new events are written to the partition.

what will happen after the Kafka consumer reads the whole partion?

1 Answers1