0

I created a cluster with 2 brokers in MSK but I cant connect using kafkajs client. I keep getting this connection timeout error

{"level":"ERROR","timestamp":"2021-07-29T13:35:35.625Z","logger":"kafkajs","message":"[Connection] Connection timeout","broker":"{broker1}","clientId":"order-app"} {"level":"ERROR","timestamp":"2021-07-29T13:35:35.627Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection timeout","retryCount":0,"retryTime":251}

For more information in cluster config I used Access control method of none and TLS encryption is enabled (plain text is not). I tried ports 9092 and 9094. I set the inbound rule to all traffic just for testing and it still seems to timeout. Also tried with ssl enabled and disabled but same results.

const { Kafka } = require('kafkajs')
    
const kafka = new Kafka({
  clientId: 'order-app',
  brokers: [
    'broker1:port',
    'broker2:port',
  ],
  ssl: true,
})
lloiacono
  • 4,714
  • 2
  • 30
  • 46
Leutrim Osmani
  • 1
  • 1
  • 1
  • How are you providing SSL certificates to your client? – OneCricketeer Jul 29 '21 at 17:26
  • There arent any, I set access control method to none so anyone can connect. Should I create the cluster using one of the methods ? – Leutrim Osmani Jul 30 '21 at 07:03
  • I've never used MSK, so I don't know what "control method" means. If you set `ssl: true` in the client, you're probably going to need an SSL-enabled broker (which you say you have) and at-least PEM files for the clients or [IAM roles](https://kafka.js.org/docs/configuration#aws-iam-example) – OneCricketeer Jul 30 '21 at 14:27
  • is your kafkajs client running in the same VPC where MSK is deployed? – Achyut Vyas Aug 04 '21 at 16:26
  • Hello! did you fix your problem? – quit Jul 10 '23 at 07:59

1 Answers1

0

This is most likely a security group issue.

If you are using EC2 consumers, ensure they are in the same VPC, and their security group allows access to the MSK cluster.

If you are using Lambda Consumers with an MSK Trigger, it gets a bit tricky. The MSK trigger uses the security group of your cluster, not the security group of your Lambda function (contrary to what you would assume). We had to speak with a subject matter expert with AWS to figure out that tidbit. Due to this configuration, you must ensure your MSK cluster security group also has an ingress rule that allows traffic from itself (weird, right?).

Borduhh
  • 1,975
  • 2
  • 19
  • 33