3

I am trying to deploy a rails application on Amazon's Elastic Beanstalk. I am not very well-versed in network/server environments, so I picked up some knowledge along the way by reading documentations and researching whatever technologies are present in my stack, so bear with me if I make any mistakes.

So here's what I am trying to achieve:

I want to deploy my Rails application with Docker on Elastic Beanstalk with my instances launched in a VPC with two public facing subnets and an associated RDS instance. So far so good, I got the stack set up, but upon deployment I get an error which indicates that my EC2 instance is not able to connect to the Postgres RDS instance:

[91mrake aborted! [0m[91mPG::ConnectionBad: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

So, in essence my feeling is that I got quite far with the exception being that my RDS instance cannot be reached by my EC2 instance. My guess is that there are non-satisfiable Security Groups even though I tried to make sure to add both the EC2 instance and the RDS instance into the same security group where incoming traffic on port 5432 (Postgres) is allowed by the Security Group itself.

I added a Security Group with my Ip's CIDR and could confirm that the database is reachable from my machine. That means that the connection parameters I specified in my environment are correct and that it's not related to wrong credentials/configuration.

What could be the culprit for my EC2 instance not being able to establish a connection to my RDS instance ? Again, I'm almost positive on the fact that it lies within my Security Group settings, but I haven't been able to narrow down the source of the problem.

If you require any more information (which I intentionally left out for brevity) please let me know (for example security groups for VPC, EC2 or RDS, etc.).

the_critic
  • 205
  • 2
  • 11

1 Answers1

1

Since RDS requires you to have two availability zones when deploying in a VPC, you need to make sure that beanstalk is able to get to both of them via Network ACLs as well as the permissions for the instance based security groups.

Only your ELB and your NAT instance/NAT gateway need to be public subnets, everything else should be in private subnets.

Security groups are stateful and network groups are stateless so while you only need to allow inbound rules for the security groups, you need to allow BOTH inbound and outbound ports from your beanstalk subnet to both RDS subnets using Network ACLs. See Security in Your VPC.

Here is a sample eb create to create the beanstalk environment (replace square bracketed strings):

eb create [BEANSTALK_ENVIRONMENT] --instance_type m3.medium --branch_default --cname [BEANSTALK_CNAME] --database --database.engine postgres --database.version [x] --database.size 100 --database.instance db.m4.large --database.password xxxxxxxxx --database.username ebroot --instance_profile [BEANSTALK_EC2_IAM_PROFILE] --keyname [SSH_KEY_NAME] --platform "64bit Amazon Linux 2015.03 v1.3.0 running Ruby 2.2" --region us-east-1 --tags tag1=value1,tag2=value2 --tier webserver --verbose --sample --vpc.id [vpc-xxxxxx] --vpc.dbsubnets [subnet-db000001,subnet-db000002] --vpc.ec2subnets [subnet-ec200001] --vpc.elbsubnets [subnet-elb00001] --vpc.elbpublic --vpc.securitygroups [sg-00000001] --sample --timeout 3600

subnet-db000001 NETWORK ACL RULES:

Inbound: Port Range: 5432, Source [subnet-ec200001 (as ip range)], Allow
Outbound: Port Range: 5432, Source [subnet-ec200001 (as ip range)], Allow

subnet-db000002 NETWORK ACL RULES:

Inbound: Port Range: 5432, Source [subnet-ec200001 (as ip range)], Allow
Outbound: Port Range: 5432, Source [subnet-ec200001 (as ip range)], Allow

subnet-ec200001 NETWORK ACL RULES:

Inbound: Port Range: 5432, Source [subnet-db000001 (as ip range)], Allow
Inbound: Port Range: 5432, Source [subnet-db000002 (as ip range)], Allow
Outbound: Port Range: 5432, Source [subnet-db000001 (as ip range)], Allow
Outbound: Port Range: 5432, Source [subnet-db000002 (as ip range)], Allow

subnet-elb00001 NETWORK ACL RULES:

Inbound: Port Range: 80, Source 0.0.0.0/0, Allow
Inbound: Port Range: 443, Source 0.0.0.0/0, Allow
Outbound: Port Range: 80, Source 0.0.0.0/0, Allow
Outbound: Port Range: 443, Source 0.0.0.0/0, Allow

An additional note about Network ACLs -- many services don't respond on the original port but use an ephemeral port. So you may have to add the following to the inbound AND outbound network ACLs for subnets with EC2 instances:

Outbound: Port Range: 1024-65535, Source 0.0.0.0/0, Allow
Outbound: Port Range: 1024-65535, Source 0.0.0.0/0, Allow

There are also several useful scenarios in Recommended Network ACL Rules for Your VPC.

I hope this helps.

Peter M
  • 973
  • 2
  • 15
  • 27
  • I'll try that and get back to you! Thanks in advance. Would you be ready to answer any questions that might come up ? – the_critic Mar 08 '16 at 16:36
  • sure. one other note about Network ACLs. Many services don't respond on the original port but use an ephemeral port. So you may have to add the following to the inbound AND outbound network ACLs so your beanstalk instance can get to the internet for gem installs, etc: Custom TCP Rule, TCP(6), 1024-65535, 0.0.0.0/0, ALLOW. See http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_ACLs.html#VPC_ACLs_Ephemeral_Ports – Peter M Mar 08 '16 at 16:46
  • This is the correct chat room if you'd like to join: http://chat.stackexchange.com/rooms/36705/elastic-beanstalk-setup?tab=general – the_critic Mar 08 '16 at 17:28
  • I'd like to thank you for your answer, however I can't get it to work. If I set it up like you say I get issues with my load balancer saying that the underlying EC2 instances are "out of service" (due to health checks failing). Is there any way I can debug the whole stack ? (ELB -> VPC -> EC2). I'm aware of the fact that the health checks test port 80 on my EC2 instance, but for some reason I can't make the Load Balancer work (possibly due to a misconfiguration on my part). I deployed the sample app on EB, so it should be able to deploy (which it does, but then fails after status "OK" by ELB) – the_critic Mar 08 '16 at 22:30
  • If the app deploys successfully the load balancer should be able to pass the health check, which it does not. A connection timeout follows and EB enters "Severe" status. – the_critic Mar 08 '16 at 22:35
  • first confirm that your site comes if you go directly to the beanstalk instance and bypass the elb. Then make sure you have opened ports 80/443 in and out of both ELB subnets and beanstalk web subnet using network ACLs. Beanstalk should take care of the security group between the elb and the web instance. – Peter M Mar 08 '16 at 23:08
  • The funny thing is that EB won't go along with that. It enters "Severe" state as soon as the ELB fails... – the_critic Mar 08 '16 at 23:10
  • network acls for both your elb subnet and your beanstalk ec2 web subnets both allow ports 80/443 inbound and outbound to/from 0.0.0.0/0? – Peter M Mar 08 '16 at 23:26
  • Yes I do, I'm not confident in saying that I am totally aware of what I'm doing that's why I suggested taking this into the chat. The NAT, Internet Gateway and the subnet ACLs are a little over my head... I wanted to recreate the environment with you step by step, but I can imagine that this is not the route you are willing to go. I first need to wrap my head around what's wrong, but it's just depressing if you don't even know where to start ... – the_critic Mar 08 '16 at 23:42
  • I can't do a chat -- the whole process from start to finish would take hours or days that way. do you have aws support? if so, you should open a case with them. – Peter M Mar 09 '16 at 20:45
  • I think I can pull it off within an hour or two with your help. I'm pretty sure it would just require some guidance for the security groups, ACLs and possibly the routing tables. I can do it from the AWS console pretty quickly. I have thought about contacting AWS support, but I'm not sure yet if I should go that route. Have you done something like this before or are you sure you could potentially set this up with me ? I'd be willing to compensate you for your support (given that you really know what you are doing, which I'm sure you do). Should you agree with it, I would ask us to continue.. – the_critic Mar 09 '16 at 20:51
  • on a different communication channel (of your choice [Skype would come to mind...]). – the_critic Mar 09 '16 at 20:52
  • sorry -- i don't have the time to do this. contact aws support, they should be able to help you. my answer above really should be able to get your there. you just need to get the network ACLs and the the security groups right. use a nat gateway, don't build your own nat instance. good luck – Peter M Mar 10 '16 at 17:48
  • Alright, thank you for taking your time anyway! Really appreciate it. I cannot mark your answer for now as I'm not really in a position to verify it (maybe I will be after contacting AWS support). – the_critic Mar 10 '16 at 18:17