3

I created an ecs service running a task definition that creates a docker container running a flask app. The flask app is getting an image through the http request and returning a ML model inference.

Since I want this service to run in production env, I created an internal network load balancer, and attached it to a vpc endpoint service. The nlb is forwarding requests to a target group which registers all ECS cluster's instances.

I have an API gateway which integrates with a lambda function that is configured inside the same vpc, and from the lambda I need to access the ecs service(through the nlb or vpc endpoint)

*** However, I keep getting 'max retries exceeded' with python requests- as the endpoint is not reachable. Nor the nlb and neither the vpc endpoint service.

Please help me understand whats wrong, or suggest another infrastructure that handles these requests in a production env.

** The docker container is valid and responding using postman directly, but I need postman to send to API gateway and get an edited response from the lambda.

Attached are some of my configurations. Let me know if you think more configurations are required. Thanks:)

Lambda connection error to endpoint

VPC endpoint service configuration

NLB configurtion and integration

Lambda vpc configuration

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
xGKx
  • 65
  • 2
  • 6
  • Does the endpoint work otherwise? I want to confirm that the issue is only with lambda function? If you use EC2 instance, does the endpoint work? – Marcin Oct 13 '20 at 12:14
  • Yes, I can communicate with the instance directly, but cant get it to work through the vpc/nlb – xGKx Oct 13 '20 at 12:15
  • I would go back one step, and concentrate only on the NLB and the endpoint service. Don't use lambda nor api gateway. Just EC2 instance which you can login and try curling the endpoints from there. Once you make it work, then move to how to making it work with lambda. – Marcin Oct 13 '20 at 12:33
  • Also check with HTTPS. From screenshot it seem you are using HTTP only. I think the endpoint services operate on port 443 for HTTPS. So security groups also need to be modified accordingly. – Marcin Oct 13 '20 at 12:36
  • Checked both HTTP and HTTPS before. Not working. I did ssh into the ec2 instance and curling the nlb generates a valid response. I just cant get the lambda to reach it, though it is configured in the VPC and has the vpcExecution role. – xGKx Oct 13 '20 at 13:07
  • Security groups for lambda? Maybe endpoint secirity group does not allow connections from lambda sg? – Marcin Oct 13 '20 at 21:42
  • how are you running your flask app? make sure it is listening on 0.0.0.0. – Asdfg Oct 14 '20 at 14:08

1 Answers1

1

Rest of your configurations seem alright, and it is validated by accessing the service inside EC2.

There is only one piece to the puzzle I can point out, i.e. while attaching your Lambda Function with the VPC, only use the private subnets. Currently, I'm not sure if the attached subnets to your Lambda Function are private or public.

amsh
  • 3,097
  • 2
  • 12
  • 26