5

I was running a serverless web application on a lambda inside a VPC, and connecting to a Aurora-MySQL RDS instance, with inbound rules to allow traffic from the security group of the lambda The connection was working fine, however, quite often the lambda cold start was giving me a timeout. After some research, I found out that running a lambda on a VPC brings an additional cost on startup and I saw the recommendation in more than 1 place to avoid using lambda on a VPC except if you strictly need to access some resource in the VPC.

So, I decided to move my RDS to a publicly accessible instance, so my lambda can access it over the internet and remove the lambda from the VPC.

So, I changed the RDS Public accessibility option to Yes and edited the security group to allow inbound connection from any IP. I have also removed the VPC from the lambda, so the lambda is not running on a VPC anymore I thought it was gonna be enough.

But then my lambda started failing to connect to the database I tried to connect using my local client, again, failure

tried pinging to the hostname, got request timeouts

After digging a bit into it, I found that my DB instance subnet group having some private subnets might be a problem (?) So, I have created a new subnet group with only public subnets, and tried to move my db instance to the new subnet group... but got this message:

You cannot move DB instance my-instance to subnet group my-new-group. The specified DB subnet group and DB instance are in the same VPC.

Ok, it seems that I can't move to a different subnet in the same VPC, I started trying to create a new VPC, but it doesn't seem to be right and I'm sure there is something else I am missing here.

I also read about Network ACL, and thought that this might be the problem, but my rules seem to be fine, with the default rule to allow any traffic (and the rule * to DENY)

ALL Traffic ALL ALL 0.0.0.0/0 ALLOW

My RDS Network settings

Subnet group
default

Subnets
subnet-11111111
subnet-22222222
subnet-33333333
subnet-44444444
subnet-55555555
subnet-66666666

Security
VPC security groups
default (sg-111111)
( active )

Public accessibility
Yes

My Security group inbound rules

Type Protocol Port range    Source  Description - optional
All traffic All All 0.0.0.0/0   -
All traffic All All ::/0    -

Still can't connect, can't connect with my local client, can't even ping it:

Connecting through my local client

Can't connect to MySQL server on 'my-instance.xxxxxxxxxx.us-east-1.rds.amazonaws.com' 

ping my-instance.xxxxxxx.us-east-1.rds.amazonaws.com
PING ec2-xx-xx-xx-xx.compute-1.amazonaws.com (xx.xx.xx.xx): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2

Any idea of what I am missing here?

UPDATE

My VPC has internet access (I can access internet services from it, not an issue), I have an Internet Gateway and NAT Gateway in place.

I'm using Zappa for the lambda deployment, which takes care of creating a keep-warm function... however, I know that concurrent requests could still be an issue

The issue with VPC in lambda is that it can add 10s on the cold start, which is a no-deal for some of my use cases: https://www.freecodecamp.org/news/lambda-vpc-cold-starts-a-latency-killer-5408323278dd/

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
dfranca
  • 5,156
  • 2
  • 32
  • 60
  • 3
    The "cold start in a VPC" issue has been fixed by AWS (see [Announcing improved VPC networking for AWS Lambda functions | AWS Compute Blog](https://aws.amazon.com/blogs/compute/announcing-improved-vpc-networking-for-aws-lambda-functions/)). It is unlikely you will need to do all that pre-warm stuff now. – John Rotenstein Mar 20 '20 at 11:34
  • @JohnRotenstein Amazing! Thanks – dfranca Mar 20 '20 at 13:25

2 Answers2

2

Besides enabling "public access" on your RDS instance, you need to enable internet to the VPC using an internet gateway. After you attach it to the VPC, you need to route the data from the database subnets to the internet gateway. Check here

But I would not advise you expose your database like this. If you are having issues with lambda cold start, you should create an event to keep it warm.

Stargazer
  • 1,442
  • 12
  • 19
  • Check this instruction also @danielfranca – Avinash Dalvi Mar 20 '20 at 09:26
  • Thanks for your answer, but I have already an internet gateway on my VPC, I can access internet services from it already... and I'm using Zappa for the lambda deployment, which sets a keep warm function – dfranca Mar 20 '20 at 10:32
  • And as I mentioned, I can't access it even from my local client – dfranca Mar 20 '20 at 10:54
  • @danielfranca I run several services using zappa connected to private databases, and I never had issues with timeout caused by could start. Maybe you have requests taking longer than 30 secods to be served. You function will never take more than 30 seconds to be provisioned. Specially with python, your function will be ready on 5 seconds tops. If you are facing timeout, I don't think it has nothing to do with cold start. – Stargazer Mar 20 '20 at 12:42
  • And again, you have to route your databases subnets to the internet. – Stargazer Mar 20 '20 at 12:46
  • Creating a custom route table, targetting an internet gateway as the destination and 0.0.0.0/0 as source .... and THEN do the associations with the subnet of the RDS. Took me half a day to figure this out, there is no end-to-end tutorial out there – clunven Aug 07 '23 at 13:02
1

Things you need to is :

  • Create new subnet group with default VPC
  • assign two subnet for availability zone
  • then modify your RDS instance
  • change subnet group to newly created group
  • mark "Publicly accessibility" to Yes.
  • Check your VPC is using internet gateway or not.

Check lambda security group whether it's open for outbound connection for Database port is not available or not.

No need to create different VPC for RDS. Use Default VPC.

As recommended by @stargazer try to not to expose publicly or out of VPC. Its works well inside VPC.

Avinash Dalvi
  • 8,551
  • 7
  • 27
  • 53
  • 1
    AWS doesn't allow to move to a different subnet group on the same VPC: https://serverfault.com/questions/816820/aws-can-not-change-db-subnet-group-for-aws-rds – dfranca Mar 20 '20 at 10:37
  • The issue is that VPC cold start can add up to ~10s itself https://www.freecodecamp.org/news/lambda-vpc-cold-starts-a-latency-killer-5408323278dd/ – dfranca Mar 20 '20 at 10:38
  • you can modify subnet group. you have mark as immediate changes while doing modification. – Avinash Dalvi Mar 20 '20 at 10:41
  • That is what happens when I try to modify for a different subnet group in the same VPC: https://imgur.com/a/ZYanscA – dfranca Mar 20 '20 at 10:47
  • i am trying to say is create subnet group with default vpc then you can able to do. error message also saying the same thing which i explained. – Avinash Dalvi Mar 20 '20 at 11:30