How to implement a secured kafka environment with a public interface?

Question

Scenario: Kafka cluster, secured with SSL encryption and SASL/PLAIN authentication, resides in an AWS private subnet within a dedicated VPC. Within the private subnet, all is fine. I use self-generated CA and keys to secure the communication, host identities are based on the AWS internal DNS.

What I want to achieve: to have a possibilty to access the producer API (not REST API) from the outside.

I am struggeling with the combination of keys, DNS, kafka listeners and the fact that the permanent connection from a producer to a broker is possibly not the one that I used to start the connection.

Several attempts with reverse proxies failed - even a ssh tunnel does not work since the keys cannot be resolved.

Does anybody have a kind of reference architecture for such a case? I spare the config details here since it is too much spread in the different configs, keys, etc. but if needed I can provide my settings.

Answer 1

Ok I found a solution - maybe not most efficient but fine for me. I implemented three level 4 load balancers in front of the three brokers and configured each broker's advanced_listener address with the one of the load balancer. works like a charm - even if it's comparatively expensive and throttles traffic rates somewhat.