Kafka message key is null using Confluent.Kafka consumer library

Question

I am using Debezium to perform CDC against my DB to create messages in Kafka. Using tools like kafdrop and OffsetExplorer I am able to see the message key and value. However, from a .NET framework application, using the Confluent.Kafka libraries, when I consume the messages, the message key is always null.

How can I retrieve the message key using the Confluent.Kafka library?

Here is the code for the VS project:

using System;
using Confluent.Kafka;
using System.Threading;

namespace KafkaConsumer
{
    class Program
    {
        static void Main(string[] args)
        {
            var config = new ConsumerConfig
            {
                BootstrapServers = "kakfa:9092,localhost:9093",
                GroupId = "foo",
                AutoOffsetReset = AutoOffsetReset.Earliest,
                
            };

            var topics = "CAC_connector.dbo.sessionLogs";
            bool cancelled = false;

            // Define the cancellation token.
            CancellationTokenSource source = new CancellationTokenSource();
            CancellationToken cancellationToken = source.Token;

            using (var consumer = new ConsumerBuilder<Ignore, string>(config).Build())
            {
                consumer.Subscribe(topics);

                while (!cancelled)
                {
                    var cr = consumer.Consume(cancellationToken);
                    Console.WriteLine($"Consumed record with key {cr.Message.Key} and value {cr.Message.Value.Substring(0, 96)}");
                }

                consumer.Close();
            }
        }
    }
}

Cheers, Kyley

I hadn't noticed that when copying the code from another example. I was looking for configuration settings and didn't understand that constructor params. Thank you for pointing that out to me. When changing to `ConsumerBuilder — Kyley Jex, Jul 21 '21 at 17:26

score 3 · Answer 1 · answered Jul 21 '21 at 17:33

3

As OneCricketeer pointed out, I was initially constructing the ConsumerBuilder to Ignore the key. When I changed the constructor to ConsumerBuilder<string, string> the key is included in the Message.

answered Jul 21 '21 at 17:33

Kyley Jex

53
8

Kafka message key is null using Confluent.Kafka consumer library

1 Answers1