1

Has anybody successfully established client connection to Amazon MSK Kafka cluster using JavaScript? No YouTube video or online example AFAIK is out there. Attempts to use KafkaJs npm module are not working for me, because the SASL AWS I am roles is not supported without installing IamAWSLogin plugin on the brokers which you can’t ssh into.

Trying to use plain SASL method doesn’t work on KafkaJs because aws doesn’t use username and password.

I am not finding kafka-node useful as well.

Any leads?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Fidelis Chimombe
  • 11
  • 1
  • 2

3 Answers3

2

There is a new feature in development that permits to inject mechanisms for auth with AWS.

https://medium.com/@jm18457_46341/using-amazon-msk-with-iam-access-control-for-apache-kafka-and-node-js-with-kafkajs-71638912fe88

Maybe is necessary to add a branch dpendency for your project, and it is a risk for production builds, however the good news is was reviewd and shoudl be merged soon :)

https://github.com/tulios/kafkajs/pull/1101

Eduardo Fabricio
  • 2,151
  • 2
  • 25
  • 32
1

We've battled with IAM too, and it seems to be for Java clients only.

We have got it working with username/password. Details for MSK config are here https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.html. I recommend when you set up MSK using a custom security group and setting up appropriate inbound access for the MSK ports.

When the cluser is set up, use the "View client information" button to get the brokers/ports to use.

Then this is your KafkaJS client setup:

new Kafka({
  clientId: 'my-app',
  brokers: ['something.kafka.us-east-1.amazonaws.com:9096', 'somethingelse.kafka.us-east-1.amazonaws.com:9096'],
  ssl: true,
  sasl: {
   mechanism: 'scram-sha-512',
   username,
   password,
  }
})
JimmyDix
  • 81
  • 4
  • 1
    This unfortunately didn't work for me. Is there any special config that needs to happen in the MSK cluster to enable `scram-sha-512` to work? – itajenglish Mar 15 '23 at 19:25
1

I was able to connect and use Amazon MSK Kafka cluster, via kafkajs library. Initially I followed instructions found in docs of kafkajs library on how to use aws mechanism for sasl.

Considering that by default MSK Kafka cluster is not accessible from internet, I created a VPN client first following this video: https://www.youtube.com/watch?v=Bv70DoHDDCY, made sure that the client authorized users to access subnets of my VPC and after that I simply removed the sasl part from configuration.

so... I used something like:

const kafkaClient = new Kafka({
  clientId: 'local-client',
  brokers: [
    'b-2.xxx.xxx.xx.xxx.xx.eu-central-1.amazonaws.com:9094',
    'b-3.xxx.xxx.xx.xxx.xx.eu-central-1.amazonaws.com:9094',
    'b-1.xxx.xxx.xx.xxx.xx.eu-central-1.amazonaws.com:9094'
  ],
  ssl: true,
})

If sasl: {...} part would be there, I would get weird errors like "[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Request is not valid given the current SASL state"

Most probably sasl is not needed anymore because of the VPN connection.

CRK
  • 1,299
  • 12
  • 9