How do I make the C# Kafka client consume the latest topic message?

Question

I created a C# Kafa client with Confluent.Kafka, and I have been trying to read the latest message only from two different topics, but it reads all messages, not just the latest one.

Here is my kafka client class, I'm not sure what I'm missing here.

public class KafkaClient
{
    private readonly string _bootstrapServers;
    private readonly string _clientId;
    private readonly string _groupId;

    private readonly Dictionary<string, string> _topicDictionary = new();

    public KafkaClient(IOptions<KafkaSettings> kafkaSettings)
    {
        _bootstrapServers = kafkaSettings.Value.BootstrapServers;
        _clientId = kafkaSettings.Value.ClientId;
        _groupId = kafkaSettings.Value.GroupId;

        _topicDictionary[TopicConstant.BetTopic] = kafkaSettings.Value.Topic.BetTopic;
        _topicDictionary[TopicConstant.UserTopic] = kafkaSettings.Value.Topic.UserTopic;
        _topicDictionary[TopicConstant.WalletTopic] = kafkaSettings.Value.Topic.WalletTopic;
    }

    public List<string> Consume(string[] topics, CancellationToken cancellation)
    {
        List<string> messages = new();

        var config = new ConsumerConfig
        {
            BootstrapServers = _bootstrapServers,
            GroupId = _groupId,
            AutoOffsetReset = AutoOffsetReset.Latest,
            EnableAutoCommit = true
        };

        using var consumer = new ConsumerBuilder<Ignore, string>(config).Build();

        foreach (var topic in topics)
        {
            if (!_topicDictionary.TryGetValue(topic, out var topicValue))
                throw new Exception($"{topic} Does not Exist");

            consumer.Subscribe(topicValue);
            var consumeResult = consumer.Consume(cancellation);
            messages.Add(consumeResult.Message.Value);
            consumer.Commit(consumeResult);
        }
        consumer.Close();

        return messages;
    }
}

Thank you

The parameters to a Cosume() method has "string[] topics" which is an array. You are printing all the messages in the array. You need to pass only the latest to the method. — jdweng, May 04 '23 at 09:20
@jdweng I loop through the topic and I read each topic here consumer.Subscribe(topicValue); but this reads all messages in the topic although I configured the consumer to read the latest message AutoOffsetReset = AutoOffsetReset.Latest, — Ahmed Elshorbagy, May 04 '23 at 11:19
You did not post the code that calls the Comsume() method. The code that does the call needs to filter for the latest instead of sending the entire list. — jdweng, May 04 '23 at 12:10
`AutoOffsetReset.Latest` already defaults to read from the end of the topic. Remove the loop and consume one event to get the next, latest one producer may send — OneCricketeer, May 04 '23 at 12:34

score 0 · Answer 1 · answered May 05 '23 at 12:51

I solved it by adding a key to the message while producing it

var message = new Message<int, string>
{ 
   Key = 1, 
   Value = JsonSerializer.Serialize(user),
   Timestamp = new Timestamp(DateTime.Now) 
};

And The second message the key value will be 2 and so on.

How do I make the C# Kafka client consume the latest topic message?

1 Answers1