I am trying to setup multi-region kafka clusters that can easily scale by adding additional brokers and additional clusters. To avoid the producers having to worry about addition of new clusters, its there a way to expose Kafka using a single (or a few fixed) endpoints so the end users don't have to concern themselves with changes being made in the background.
Currently the setup relies on the AWS MSK provisioned service and I am trying to follow the setup described here https://aws.amazon.com/blogs/big-data/how-goldman-sachs-builds-cross-account-connectivity-to-their-amazon-msk-clusters-with-aws-privatelink/. Rather than exposing to another account in AWS, would it be possible to expose to the internet using a public domain with multiple clusters sitting behind a URL. I am thinking if it would be possible to control traffic to a cluster using Route 53 where I can load balance as required (e.g. 100/0 or 50/50). So far I have setup a NLB with a target group (with health checks enabled) but have not been able to send any events to Kafka thus far.
Please help me understand:
- If this setup is even possible
- What are the best alternatives if not (I do not want to use mirror maker to replicate data across 2 clusters as it does not solve the issue of abstracting the client from the underlying clusters).
The fallback for me is to add brokers from every cluster in the application and inform teams managing those applications when additional clusters are added.
