I am trying to create Kafka producer and consumer with SASL authentication. During SASL authentication I would like to know about the specific broker and port that is selected for very first time from the list of brokers passed. I only know hostname but not port number.
Asked
Active
Viewed 284 times
0
-
What is the specific problem you are trying to solve? Clients connect to _multiple brokers_, based on where partitions are hosted – OneCricketeer Jun 01 '22 at 18:07
-
So, I would like to change the broker servername based on port picked up during producer.send(). Since it is selected at last min, I dont know how to find which broker will be picked and where I can intercept to change the broker servername. – stackuser1 Jun 04 '22 at 00:36
1 Answers
0
only know hostname but not port number
A port is required by bootstrap-server config, so I'm not sure I understand this...
the specific broker and port that is selected for very first time from the list of brokers passed
The one that is selected shouldn't matter. All brokers can respond to the same request, and all broker info will be returned to the client before the producer starts. You can use AdminClient.describeCluster for cluster information.
But sounds to me like you need to talk to the kafka admin who configured advertised.listeners.
You can also use kcat -L option to find this information remotely.
would like to change the broker servername based on port picked up during producer.send()
You cannot modify data in a Producer batch. The advertised listeners are what will be used by the network protocol.
You'd need to implement a proxy that would redirect the data somewhere else or intercept the Kafka TCP packets and rewrite them to do anything else.
In Java, to find the exact server a particular record would be sent, you'd need to use the DefaultPartitioner class to find the partition each record would be sent to, using its key. This assumes your records have non null keys.
Then you need to describeTopics using an AdminClient instance. The result of describing a topic returns a list of TopicPartitionInfo. Filter this by the partition gotten for the record.
These have a leader() method that returns a Node object, which has host and port information. The leader partition is where the data will be sent when the producer buffer is flushed.
https://kafka.apache.org/31/javadoc/org/apache/kafka/common/Node.html
For a consumer, look at its partition assignment, then do the same.
OneCricketeer
- 179,855
- 19
- 132
- 245
-
-
For example, you could write a REST application that accepts HTTP requests and wraps a Producer but at startup, it could cache Kafka cluster details – OneCricketeer Jun 13 '22 at 12:54
-
I was trying to put load balance in front to it for redirection but looks like I need custom option – stackuser1 Jun 13 '22 at 22:45
-
TCP load balancers will not work with Kafka. Client must communicate directly to individual leader brokers. – OneCricketeer Jun 13 '22 at 22:54