I have a use case in which I have a Kafka producer that needs to talk to a Kafka broker and a consumer, but with the added bonus of both the Broker and the Consumer being inside a customer's VPC (AWS VPC).
How I have been looking at ways to get messages through and I have been considering different tunneling technologies (geneve, ssh tunnels etc).
The problem with all of these approaches is that once I configure a tunnel I have to "override" the Kafka broker code such that it redirects messages to the tunnel IP and not the broker IPs.
Now I tried researching a bunch of things before deciding to post, but so far they have been dead ends.
I was reading the Kafka protocol: https://aws.amazon.com/blogs/aws/new-managed-nat-network-address-translation-gateway-for-aws/ and while I could inject arbitrary headers in the record, without having an additional process somehow "intercepting" these messages, hacking apart the header and redirecting the message, I do not see it useful.
I was looking at the Apache Kafka client code and I found this https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/ProducerInterceptor.html but this at best overrides the records.
I was also looking at https://github.com/apache/kafka/blob/4090f9a2b0a95e4da127e4786007542276d97520/clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java but the Producer config does not provide overrides or any kind of control over the underlying TCP client.
I was also considering overriding the /etc/hosts file, but these tunnels would have to be dynamically created and/or destroyed and moreover the IP Addresses of the brokers are private IP addresses and there might be IP collisions for two different customers if they are on different VPCs with the same/similar CIDR ranges.
At this point, I am kind of out of options and I would appreciate some help.