How can I get a list of all `PartitionTopic`s for a topic in Confluent.Kafka?

Question

I am using confluent-kafka-dotnet (Confluent.Kafka) to produce messages for Kafka.

I would like to manually manage which partitions messages go to in order to preserve order for certain groups of messages.

How is it possible to get a list of PartitionTopic for a Kafka topic?

I have looked at this solution in Java. But I don't understand how to achieve the same functionality with Confluent.Kafka.

Alternatively, it would be acceptable to send messages with keys, because the same keys are guaranteed to be on the same partitions. But again, I could not find a way to create a new Message with any key type other than Null. So, an example of sending a message with a non-null key would be helpful.

Please share a [mcve] which includes where you are attempting to set the partition key. — mjwills, Jul 14 '20 at 09:47

score 3 · Accepted Answer · answered Jul 14 '20 at 11:36

3

To get a list of TopicPartitions for a single topic you could use AdminClient class:

using (var adminClient = new AdminClientBuilder(new AdminClientConfig { BootstrapServers = "bootstrap-servers" }).Build())
{
    var meta = adminClient.GetMetadata(TimeSpan.FromSeconds(20));

    var topic = meta.Topics.SingleOrDefault(t => t.Topic == "topic-name");

    var topicPartitions = topic.Partitions;
}

You can find more AdminClient examples here. But notice the warnings, saying that: "The API for this functionality is subject to change."

answered Jul 14 '20 at 11:36

rytisk

1,331
2
12
19

This doesn't work. This returns TopicMetadata which has PartitionMetadata. In the KafkaAdmin, the PartitionTopic is a different object. – TheAlgorist May 27 '21 at 22:05
@TheAlgorist There is no such class `PartitionTopic` in confluent-kafka-dotnet. Maybe you are referring to the `TopicPartition`? If you are, then the `PartitionMetadata` retrieved from `AdminClient` and the topic name is sufficient to construct the `TopicPartition` objects for your topic. – rytisk May 28 '21 at 16:20

score 0 · Answer 2 · answered Jul 14 '20 at 10:12

0

I have worked out that the reason I could not create messages with a non-null key was because I had specified my Producer with <Null, ...>.

Thank you to mjwills for asking me for minimal reproducible example, which prompted me to work this out.

answered Jul 14 '20 at 10:12

Mysterious Entity

53
1
6

score -1 · Answer 3 · answered Oct 27 '22 at 14:01

If you need to explicitly configure the offsets of a specific consumer for a topic, you have to set the offset PER TopicPartition. This requires having updated Metadata for the cluster. For example:

        Metadata meta = adminClient.GetMetadata(TimeSpan.FromSeconds(20));
        // Query where the offset is for each partition.
        foreach (TopicMetadata topicMeta in kafkaMetadata.Topics)
        {
            if (topicMeta.Topic == "sometopic")
            {
                for (int i=0; i<topicMeta.Partitions.Count; i++)
                {
                    TopicPartition tp = new TopicPartition(topic, topicMeta.Partitions[i].PartitionId);
                    WatermarkOffsets wo = consumer.QueryWatermarkOffsets(tp, TimeSpan.FromSeconds(60));
                    //consumer.Seek(new TopicPartitionOffset(tp, wo.High));
                }
                break;
            }
        }

I don't mind the down-voting for this not answering the question head-on, but what I was attempting to show was exactly how you construct a new TopicPartition based on the AdminClient. There isn't another example of it here, and it's not obvious that it's legal to just make a new one when needed. — Jason Hughes, Feb 13 '23 at 03:18

How can I get a list of all `PartitionTopic`s for a topic in Confluent.Kafka?

3 Answers3