0

There is AWS MSK 3.1.1 setup with 3 brokers, one in each availability zone.
There is NLB.
There is kafka-integrations-dev.com DNS name for NLB.

NLB has 3 listeners for IAM brokers:

  • TLS:7200 -> target group #1 TLS:9098 -> IP of broker #1, port 9098.
  • TLS:7201 -> target group #2 TLS:9098 -> IP of broker #2, port 9098.
  • TLS:7202 -> target group #3 TLS:9098 -> IP of broker #2, port 9098.

There is SSL/TLS certificate with kafka-integrations-dev.com domain configured for every mentioned listeners.

Is it possible to connect to MSK cluster with IAM auth by kafka-integrations-dev.com:7200, kafka-integrations-dev.com:7201, kafka-integrations-dev.com:7202 as bootstrap servers?

Now we are getting error Hostname verification failed.

I found Making AWS MSK public using NLB and IAM authentication - Hostname verification failed The author stated that connection to MSK via NLB using IAM auth was not supported in 2021.

Lesha Pipiev
  • 3,251
  • 4
  • 31
  • 65
  • 1
    It seems really odd to have a NLB in front of kafka. Your clients connect to individual brokers directly, they should not be load balanced. – jordanm Mar 08 '23 at 16:33
  • @jordanm check the article please https://aws.amazon.com/blogs/big-data/how-goldman-sachs-builds-cross-account-connectivity-to-their-amazon-msk-clusters-with-aws-privatelink/. There is pattern #2, which is used to build cross-account connectivity between client and MSK cluster. – Lesha Pipiev Mar 08 '23 at 16:48
  • 1
    That configuration has a NLB *per* broker. – jordanm Mar 08 '23 at 16:59
  • @jordanm oh you are right – Lesha Pipiev Mar 08 '23 at 20:15
  • 1
    there are two patterns. Pattern1 has NLB per broker, but pattern 2 has a listener per broker. Both patterns won't work for IAM. Will post in answer full explanation. – EdbE Mar 09 '23 at 00:09

1 Answers1

2

Provided solution with NLB will work for TLS and SASL-SCRAM, and won't work for mTLS and SASL-IAM. Since you are asking specifically about IAM, here are some details.

When you initiate SASL-IAM protocol, you are signing your request including endpoint. Since your client is using custom domain name, broker (with different domain name) will reject such request. So, even though you can terminate TLS at NLB, you cannot re-sign the request, it would break the mechanism of IAM authentication which signs the request with Role arn, and endpoint URLs.

EdbE
  • 204
  • 1
  • 4