13

I'm trying to get down and dirty with DevOps and I'm running into a health check request timed out failure. The problem is my Elastic Load Balancer sends a health check to my EC2 instance and gets a network timeout. I'm not sure what I did wrong. I am following this tutorial and I have completed all the steps up to and including "Using a Elastic Load Balancer". My EC2 instance seems to be working fine and I am able to successfully curl localhost on port 9292 from within the EC2 instance.

EC2 instance security group setup: description

Elastic Load Balancer setup: enter image description here

My target group for the ELB routing has port 9292 open via HTTP and here's a screenshot of the target in my target group that is unhealthy. enter image description here

Health check config:enter image description here

I have a VPC that my EC2 instance is a part of and my ELB is connected to the same VPC. I do not have Apache installed and I do not have nginx installed. To my understanding, I do not need these. I have a Rails Puma server running and I can send successful curl requests to the server.

My hunch is that my ELB is not allowed to reach my EC2 instance, resulting in a network timeout and a failed health check. I'm unable to find the cause for this. Any ideas? This SO post didn't help much. Are my security groups misconfigured? What else could potentially block a routing request from ELB to my EC2 instance?

Also, is there a way to view network requests / logs for my EC2 instance? I keep seeing VPC flow logging but I feel like there are simpler alternatives.

Here's something I posted in the AWS forums but to no avail.

UPDATE: I can curl the private IP of target just fine from within an EC2 instance. I don't think it's the target instance, I think it's something to do with the security group setup. I am unable to identify why though because I have basically allowed all traffic from the Load Balancer to the EC2 instance.

Sticky
  • 3,671
  • 5
  • 34
  • 58
  • Can you show us the Health Check configuration on your load balancer? – John Rotenstein Jun 07 '17 at 05:24
  • added just now! – Sticky Jun 07 '17 at 05:26
  • I don't see any issues with your load balancer and VPC/security group configuration. I'm guessing your application simply isn't returning a 200 response code for the requests at `/`. Is your application perhaps returning a 301 response for that path? – Mark B Jun 07 '17 at 15:17
  • when I `curl -I localhost:9292`, it returns a 200 so I'm confused – Sticky Jun 07 '17 at 18:53
  • Have you allowed the traffic through from load balancer to instance? Maybe try launching an instance into the load balancer vpc subnet and see if that can connect? It must be a binding/firewall/security group issue – Liam Wheldon Jun 16 '17 at 14:33
  • I think it is a binding/firewall/security group issue but I'm not sure where to start looking. The settings all seem fine. How do I "allow" traffic through" from the load balancer? I have inbound rules for network ACL on VPC as `ALL Traffic ALL ALL 0.0.0.0/0 ALLOW` so that should work right? It's the default option. – Sticky Jun 18 '17 at 00:43
  • AWS Load balancers usually have strict outbound rules as well. It looks like you have opened inbound rules on EC2. So, I'd suggest checking outbound rules on load balancer network. – Seva Jun 18 '17 at 11:07
  • Another common issue is operating system firewall on EC2. I haven't seen this issue affecting Linux instances. But on Windows, you should open server port in Windows own firewall (otherwise it will be not visible from outside of your EC2 instance no matter what security group rules say). – Seva Jun 18 '17 at 11:11
  • 1
    Are you able to `curl -I $InstanceIP:9292` from another box in the subnet as the target, and member of the same security groups as the ALB? If yes, then make sure your ALB is in the same AZ as the target box. – Raf Jun 19 '17 at 14:15
  • I was not able to successfully curl from a member of the same security group and I found out why. It was a subnet issue. I'm going to make an extensive answer delineating the problem and explaining a lot of the common terms and problems that can arise from doing this. Thank you for all the help!! – Sticky Jun 20 '17 at 07:06

1 Answers1

9

I made my mistake during the "Setup your VPC" step. I finished creating a subnet for an RDS instance. I proceeded to start an instance and the default subnet that AWS chose when I switched to my VPC was the subnet I made for my RDS, which was NOT a public subnet. Therefore, any attempts, from any EC2 instance or my load balancer, would not be able to reach it because I had only set up my public subnet to take requests.

The solution was to create a new instance and this time, pick the correct public subnet. My original EC2 instance was associated with a private subnet while the load balancer was pointing to the public subnet.

Here's a link to a hand drawn image that helped me pin point my problem, hopefully can help anyone else who's having trouble setting up. I didn't put image here directly because it's bigger than 2MB.

Glad to answer any further questions too!

Sticky
  • 3,671
  • 5
  • 34
  • 58
  • 5
    You can have an instance in a private subnet that is behind load balance. See https://aws.amazon.com/premiumsupport/knowledge-center/public-load-balancer-private-ec2/ – mitaka Sep 19 '19 at 18:25