3

I am facing an issue with exposing AWS MSK stack - to be more precisely: case of our development process is to deliver for a client, static, publicly accessible domains for MSK brokers (it is a MUST HAVE, because we are including SSL certificates in packages, that are distributed for clients and used for producing Kafka messages, so if the domain name is changed, which we would like to avoid, a lot of client packages need to be generated and redistributed again). Amazon offers only 1 option for enabling public access to MSK from Internet - exposing brokers. This is quite problematic because if something unexpected happens with the whole stack and it will be redeployed, the broker's DNS will be changed and we need to perform activities with the client packages, described above.

I have tried some solutions, for example here: https://aws.amazon.com/blogs/big-data/how-goldman-sachs-builds-cross-account-connectivity-to-their-amazon-msk-clusters-with-aws-privatelink/ but I can't get with the SSL traffic (different certificates & DNS on load balancer won't "talk" with brokers). Have you any idea how to deal with it and publicly expose brokers by using different DNS names & SSL/TLS traffic?

msmells
  • 31
  • 2

1 Answers1

1

Both this blog and the one you mentioned describe the pattern of exposing MSK broker(s) via a network load balancer (NLB). You can attach an SSL cert generated by AWS ACM to the NLB and then create an Alias DNS record on Route 53 (or CNAME on an external DNS) to point to your NLB, which should solve your problem. Hope this helps.

Alex Chadyuk
  • 1,421
  • 2
  • 10
  • I've done the setup below: - NLB & TLS listener (on port 9094) with a certificate generated & signed with AWS PCA (in example: *.xyz.kafka.eu-central-1.amazonaws.com) - Target group attached to NLB & pointing to one of the broker's IP (port 9094) address - modified advertised listeners on the tested broker (added: CLIENT_SECURE://b-2.xyz.kafka.eu-central-1.amazonaws.com:9094) - on my localhost, I created host entry which pointing custom domain to loadbalancer IP (b-2.xyz.kafka.eu-central-1.amazonaws.com -> NLB_IP) – msmells Mar 22 '23 at 16:27
  • But when I am trying to produce message to a topic, I am still getting: ERROR [Producer clientId=console-producer] Connection to node -1 (b-2.xyz.kafka.eu-central-1.amazonaws.com/xxx:9094) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) – msmells Mar 22 '23 at 16:33