0

I am trying to split a stream into multiple partition on the basis of a partition key but apparently its not working. The implementation is such that I have a Class lets say Metrices

public class Metrice {
    public string MetriceType { get; set; }
    public double MetriceValue { get; set; }
}

The metrices will be consumed asynchronously one at a time and might be having different MetriceType. What I am trying to achieve is create partition on the basis of MetriceType. What I have tried so far.

  1. Setting Message key (partitionKey) as MetricesType;
 public Message<string, string> FormatMessage(string partitionKey, string message)
 {           
   return  new Message<string, string> { Key = partitionKey, Value = message };           
 }

The messages are always published at Partition.Value= 0

  1. Partitoner class in Confluent.Kafka library, was hoping for something similar to Custom Partitoner, this link but couldn't find any .net implementation.

So my question is, is there any way to split my income messages on the basis of a property in this case MetriceType and publish them on their dedicated partitions (ordering is essential) or the only option I have is to create a Topic using AdminClient and hardcoding the partition count, or is there an alternative approach I can look into. Thanks in advance.

milo
  • 445
  • 5
  • 12
user_19240589
  • 97
  • 1
  • 14
  • 1
    How many partitions does your topic actually have? – OneCricketeer Sep 28 '21 at 14:23
  • @onecricketeere As of now only one and my key is getting hashed and partition are getting created. I was under the impression that kafka would "categorize" the incoming messages one the basis of the partition key which clearly is a wrong assumption. – user_19240589 Sep 28 '21 at 16:37
  • 1
    It hashes the key, yes, then modulo's that hash based on the topic partition count. If there is only one partition, it cannot go anywhere else... – OneCricketeer Sep 28 '21 at 16:39

1 Answers1

0

Compared to the Java SDK, you'll need to externalize your own partitioner method that returns an int.

The ProduceAsync method, for example, of the producer accepts an integer parameter for the partition you want to send the message to

Otherwise, if the topic only has one partition, then, yes, the keys will always get hashed into that

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • what do you mean by "externalize your own partitioner method". Confluent.Kafka has a ```Partitioner``` enum and a ```PartitionerDelegate```. The later expects a partitionCount which I don't want to hardcode. – user_19240589 Sep 28 '21 at 16:35
  • 1
    In Java, `Partitioner` is a class with a `int partition` method. The dotnet API doesn't have a config for `partitioner.class`, so you can "extract" this logic into your own application, to create a "custom partitioner" outside of the default. Otherwise, look at the PR that added [custom partitioners](https://github.com/confluentinc/confluent-kafka-dotnet/pull/1436) – OneCricketeer Sep 28 '21 at 16:43