0

I have a use case in which I have a Kafka producer that needs to talk to a Kafka broker and a consumer, but with the added bonus of both the Broker and the Consumer being inside a customer's VPC (AWS VPC).

How I have been looking at ways to get messages through and I have been considering different tunneling technologies (geneve, ssh tunnels etc).

The problem with all of these approaches is that once I configure a tunnel I have to "override" the Kafka broker code such that it redirects messages to the tunnel IP and not the broker IPs.

Now I tried researching a bunch of things before deciding to post, but so far they have been dead ends.

I was reading the Kafka protocol: https://aws.amazon.com/blogs/aws/new-managed-nat-network-address-translation-gateway-for-aws/ and while I could inject arbitrary headers in the record, without having an additional process somehow "intercepting" these messages, hacking apart the header and redirecting the message, I do not see it useful.

I was looking at the Apache Kafka client code and I found this https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/producer/ProducerInterceptor.html but this at best overrides the records.

I was also looking at https://github.com/apache/kafka/blob/4090f9a2b0a95e4da127e4786007542276d97520/clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java but the Producer config does not provide overrides or any kind of control over the underlying TCP client.

I was also considering overriding the /etc/hosts file, but these tunnels would have to be dynamically created and/or destroyed and moreover the IP Addresses of the brokers are private IP addresses and there might be IP collisions for two different customers if they are on different VPCs with the same/similar CIDR ranges.

At this point, I am kind of out of options and I would appreciate some help.

Pete Ythong
  • 305
  • 5
  • 13
Arunav Sanyal
  • 1,708
  • 1
  • 16
  • 36
  • As long as you can make the tunnel resolve the advertised listeners of the brokers, it'll work – OneCricketeer Feb 10 '20 at 14:34
  • Or you could use the REST Proxy, for example, and just expose an HTTP endpoint – OneCricketeer Feb 10 '20 at 14:35
  • so lets say I create an ssh tunnel. If my tunnel resolves to the listener and i publish to the source point of the ssh tunnel, the packets should go through right? – Arunav Sanyal Feb 10 '20 at 18:57
  • Maybe not "resolves"... I would say the packets would be "routable". But you can set an advertised listener to `localhost:29092`, for example, then SSH tunnel a dynamic port forward to that broker (using remote IP), then you should be able to at least produce and consumer from `localhost:29092`. The only problem with that is that each broker would require a different port mapped to your local machine – OneCricketeer Feb 10 '20 at 20:25

0 Answers0