0

Test Result Picture When I use confluent.kafka-net, and I want to ensure the order of kafka message in .netcore, I create a mschannel,

_msChannel = Channel.CreateUnbounded();

when listening, I got a consume result and a received message, and

if (_msChannel.Writer.TryWrite(receivedMessage))
{
    _consumer.Pause(new[] { consumeResult.TopicPartition });
    // _consumer.StoreOffset(consumeResult);

    // Console.WriteLine("Pause:" + sw1.ElapsedTicks + " ,Partition:" + consumeResult.TopicPartition.Partition);
}

And when the message has no error, I use message.Ack() as follows:

public ValueTask Ack()
{
    try
    {
        _consumer.StoreOffset(_offset);
    }
    finally
    {
        var sw = Stopwatch.StartNew();
        try
        {
            _consumer.Resume(new[] { _offset.TopicPartition });
        }
        catch (Exception e)
        {

        }
        sw.Stop();

        Console.WriteLine("Resume:" + sw.ElapsedTicks + " ticks" + " ,Partition:" + _offset.TopicPartition.Partition);
    }

    return default;
}

The result is that kafka's consumer is very slow, and I found that the reason for that is the partition consume and resume, costs too much time, but I have no better solution to deal with this situation.

in my server test, the consumer consumes ~300 records per minute.

milo
  • 445
  • 5
  • 12
Boi Fox
  • 45
  • 5
  • it's hard to say, first the info is not clear, and others can't get it clearly. – Boi Fox Oct 14 '21 at 10:18
  • can you explain your use-case of pausing and resuming the consumer? – milo Oct 21 '21 at 08:00
  • In order to improve concurrency without destroying the order of messages，I used a channel (C#, "System.Threading.Channels"), When a message is obtained from a partition, and the message is stored in the channel, and then pause the partition, when the message from the channel was successfully consumed, resume the partition. – Boi Fox Oct 27 '21 at 03:12
  • Probably like what I said above. – Boi Fox Oct 27 '21 at 03:13
  • what is your delivery guarantee requirement (exactly once, at least once, at most once)? – milo Nov 03 '21 at 11:26
  • It will be at least once. – Boi Fox Nov 05 '21 at 01:44
  • you say you don't want to destroy the order of messages but the order of produced messages is guaranteed when there are produced in one partition [source](https://blog.softwaremill.com/does-kafka-really-guarantee-the-order-of-messages-3ca849fd19d2) – milo Nov 12 '21 at 08:51

0 Answers0