What does Deserialization of key mean in Consumer class in Confluent Kafka?

Question

The confluent kafka documentation says, a Consumer class is defined as follows:

Class Consumer<TKey, TValue> 

The consumer class above implements a high-level Apache Kafka consumer (with key and value deserialization).

I understand the TKey and TValue are for deserializing the key, which is sent in from the producer. For example, something like

Sending in a key from the producer would look as

var deliveryReport = producer.ProduceAsync(topicName, key, val);

Receiving the string key on the consumer end would look as

using (var consumer = new Consumer<Ignore, string>(constructConfig(brokerList, false), null, new StringDeserializer(Encoding.UTF8)))
{
    consumer.Subscribe(topics);

    Console.WriteLine($"Started consumer, Ctrl-C to stop consuming");

    var cancelled = false;
    Console.CancelKeyPress += (_, e) => {
        e.Cancel = true; // prevent the process from terminating.
        cancelled = true;
    };

    while (!cancelled)
    {
        Message<Ignore, string> msg;
        if (!consumer.Consume(out msg, TimeSpan.FromMilliseconds(100)))
        {
            continue;
        }

        Console.WriteLine($"Topic: {msg.Topic} Partition: {msg.Partition} Offset: {msg.Offset} {msg.Value}");
    }
}

Since we are passing in a key, the Consumer is initialized as

Consumer<Ignore, string>

and the message is initialized as

Message<Ignore, String>

After all that, my question is, what does deserialization of the key really mean? And why do we need to do that? Also, why do we need to pass in a key-value pair Ignore, String for performing deserialization?

Answer 1

why do we need to pass in a key-value pair Ignore, String for performing deserialization?

You don't need to pass those particular settings. You need to match the settings of the producer. Or, if you're unsure, you would give byte array object for both key and value.

If the producer didn't send a key, such as null, there is nothing to deserialize. I assume that's what the Ignore class is for. Notice you didn't provide a key Deserializer class, but did for the value

null, new StringDeserializer(Encoding.UTF8))

All Kafka messages contain key, value pairs only as bytes. The Producers use serializers, and as a consumer, you need to deserialize. Ideally, you deserialize messages into actual objects, such as strings or JSON objects or Avro, Protobuf, etc. whatever.

By default, the keys are what determines what partitions of a topic the messages you'll be consuming from originated. A null key will be equally distributed across the topic. Otherwise, the producer application can define their own partitioners and send data wherever their logic decides