0

I have two broker and I want to restrict 2nd one with particular messages for particular region.
for example we have multiple region in world to provide our services like china, India
I just wanted to divert specific partner data for India region to broker1 and china region to broker2.
If this possible, I'm looking for solution need to implement in C#, but welcome ideas.

I have initial configuration like this

`"Kafka": {`
    `"Address": "[\"localhost\"]",
`   ` "BrokerHosts": "[\"localhost:9092\"]",`
    `"BrokerHostsIndia": "[\"localhost:9094\"]",`
`}`
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
akshay verma
  • 23
  • 4

2 Answers2

1

Not really sure I understand the problem. Your config here is for clusters, not brokers.

You can do var bootstrapIndia = config.get("Kafka").get("BrokerHostsIndia"), then set that in ProducerConfig to get a producer for the "India region" cluster.

One producer cannot send to multiple clusters at once, so you need to have some Dictionary<string, IProducer>, then create multiple instances per-region.

Or produce to one region, then use tools like MirrorMaker to mirror to another.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
0

Here's how you can implement this in C#:

  1. You can create two subjects, one for India and one for China, and define the number of partitions for each. For example, you may run the following command to establish a topic called 'partner-data-india' with three partitions.

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 3 --topic partner-data-india

  1. Create a custom partitioner:

Kafka's DefaultPartitioner distributes messages round-robin. You can use a custom partitioner to partition messages by area. Hash functions can assign messages to partitions by region.

Here's an example of a custom partitioner in C#:

public class RegionPartitioner : IPartitioner
{
    public int Partition(string topic, object key, byte[] keyBytes, object value, byte[] valueBytes, Cluster cluster)
    {
        // Extract the region from the message key
        string region = key.ToString().Split('-')[0];
        
        // Assign the partition based on the region



        switch (region)
        {
            case "india":
                return new Random().Next(3); // India topic has 3 partitions
            case "china":
                return new Random().Next(4, 7); // China topic has 3 partitions, starting from partition 4
            default:
                throw new ArgumentException($"Invalid region: {region}");
        }
    }

    public void Dispose()
    {
        // Nothing to dispose
    }
}
  1. Configure the producer to use the custom partitioner:

You can set the PartitionerType property of the ProducerConfig to the fully-qualified name of the custom partitioner class. For example:

{
    BootstrapServers = "localhost:9092",
    PartitionerType = typeof(RegionPartitioner).AssemblyQualifiedName
};
  1. Produce messages with the region as the message key:

When producing messages, set the key parameter to a string that includes the region. For example:

var producer = new ProducerBuilder<string, string>(config).Build();

// Produce a message for India region
var message1 = new Message<string, string>
{
    Key = "india-123",
    Value = "Some partner data for India"
};
producer.ProduceAsync("partner-data-india", message1);

// Produce a message for China region
var message2 = new Message<string, string>
{
    Key = "china-456",
    Value = "Some partner data for China"
};
producer.ProduceAsync("partner-data-china", message2);

With this setup, messages with the region "india" in the message key will be assigned to partitions 0, 1, or 2 of the partner-data-india topic.

We hope this will help to solve the problem!

Memphis
  • 7
  • 2