0

I'm trying to use an MSK Lambda trigger with a Lambda that is connected to a VPC.

However, when deploying the Lambda I'm always getting this error:

Last processing result: PROBLEM: Connection error. Please check your event source connection configuration.

What I've tried:

  • The lambda has all the required permissions as listed in https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html
  • The lambda is inside the same VPC as the MSK cluster, I even deployed another lambda that can successfully produce messages in Kafka, with the exact same VPC configuration.
  • The subnets have a NAT Gateway attached and an internet gateway
  • The security group allows all incoming/outgoing traffic in all ports

What else can I try? What could be missing? I'm using this integration from the serverless framework.

Edit: alright, the problem is the security group of the MSK cluster.

The docs mention that you need this rule:

Inbound rules – Allow all traffic on all ports for the security group specified as your source.

But is there a way to be more restrictive rather than allowing traffic from everywhere?

Carlos Martinez
  • 4,350
  • 5
  • 32
  • 62
  • you can implement an API Key for the endpoint, you can implement BasicAuth and you can use Cognito by registering user roles that you can use to restrict access to the endpoint by role. There are many options. depends what path you want to take. lmk where you want to go with it and I can show you a few ive used if you want the help. – lopezdp Nov 25 '20 at 17:51
  • It is not an API endpoint though, it's a lambda trigger for MSK. When the event source mapping is setup between Lambda and MSK it creates a Kafka consumer group, but it looks like it's internally managed by AWS and not within the same lambda security group. I'd like to know if I can create an inbound rule in the MSK security group that allows access only to this consumer rather than "allowing access for all traffic on all ports". – Carlos Martinez Nov 25 '20 at 17:59

0 Answers0