AWS Lambda times out when calling RDS Serverless

Question

I have a VPC with two ISOLATED subnets, one for my RDS Serverless cluster, and one for my Lambda functions.

But my Lambda functions all timeout when they're calling my RDS.

My question is; is this VPC + isolated subnets a working structure for API Gateway -> Lambda -> RDS, or am I trying something impossible?

Lambda:


import * as AWS from 'aws-sdk';

const rdsDataService = new AWS.RDSDataService();

const query = `SELECT * FROM information_schema.tables;`;

export const handler = async (event) => {
  const params = {
    secretArn: `secret arn`,
    resourceArn: "rds arn",
    sql: query,
    database: 'db name'
  };
  const res = await rdsDataService.executeStatement(params).promise();
  return { statusCode: 200, body: {
    message: 'ok',
    result: res
  }};
};

My RDS and Lambda share a Security Group in which I've opened for ALL traffic (I know this isn't ideal) and my Lambda has a role with Admin Rights (also not ideal) but still only times out.

I can query the database using these parameters from the Console btw. — Jesper Bylund, Oct 15 '20 at 13:37
Please describe what you mean when you say "isolated subnets". — Mark B, Oct 15 '20 at 13:47
Let's assume the lambda does timeout always (so the Aurora cold start is excluded), it implies a misconfiguration on network level. Is the Lambda attached to the correct VPC/subnet? Any updates in the subnet policies, routing tables? You may try to enable the VPC Flow Log to see where are the packets coming from and where are they routed to or denied. — gusto2, Oct 15 '20 at 13:52
@MarkB Subnet Type "Isolated". It's one of three option: Public, Private, Isolated. https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-ec2.SubnetType.html — Jesper Bylund, Oct 16 '20 at 08:15
@gusto2 Excellent suggestion, I didn't know about VPC flow log. I believe there's a silent connection error in my rdsDataService connection, but I didn't have a way to debug. — Jesper Bylund, Oct 16 '20 at 08:16
@jarmod I've gone through it several times. Does not provide any insight. — Jesper Bylund, Oct 16 '20 at 08:17
I've never heard of any "silent connection error". If you can connect from the console, then there is something misconfigured on your VPC or routing :/ Does your rds has a piblic IP? If so, how does lambda resolve the rds hostname? could the lambda reach it? — gusto2, Oct 16 '20 at 08:40
The key thing here is that you're using the Data API with Aurora Serverless. So you don't need to make a connection from Lambda to the database in the usual fashion. The request needs to reach the RDS API endpoint, per Mark's answer. — jarmod, Oct 16 '20 at 12:15
@gusto2 Silent here meaning I could not find a way to debug it. It was timing out, because it couldn't connect. — Jesper Bylund, Oct 17 '20 at 06:31
@jarmod Thank you! I had never heard of this limitation, I'm completely baffled by it. — Jesper Bylund, Oct 17 '20 at 06:32
The Data API is actually quite a significant feature. It converts the old-style persistent TCP connection solution into something akin to a REST/HTTP request. This is really valuable in a serverless environment. No more need for connection pooling or persistent connections, plus it leverages IAM and Secrets Manager for authentication with each request. — jarmod, Oct 17 '20 at 12:34
@jarmod I agree. I hope to end up using DATA API, just surprised this wasn't better documented. Up until now I thought it was a connection library. — Jesper Bylund, Oct 17 '20 at 15:18

score 4 · Accepted Answer · answered Oct 16 '20 at 12:08

4

You are using the Aurora Serverless Data API. The API does not exist inside your VPC. You have chosen isolated subnets, which have no access to anything that exists outside your VPC. You will either need to switch to private subnets, or add an RDS endpoint to your VPC.

answered Oct 16 '20 at 12:08

Mark B

183,023
24
297
295

Thank you for connecting the dots. It's too early in the morning here ;-) – jarmod Oct 16 '20 at 12:15
1

@jarmod I totally missed the Data API thing in the question until you pointed it out. Teamwork! – Mark B Oct 16 '20 at 12:17
This is baffling.. I spent days looking at documentation and I never would've come across this. Thank you! – Jesper Bylund Oct 17 '20 at 06:18

score 3 · Answer 2 · answered Apr 23 '21 at 02:32

It is important to call out that RDS API != RDS Data API; the two are different. You use the RDS API for standard RDS instances; for something like Aurora Serverless, you use the RDS Data API.

For anyone running across this in the future, there is now some helpful documentation describing how to create an Amazon VPC endpoint to allow access to the RDS Data API.

If you're using Terraform to create the VPC endpoint, here's a snippet that essentially replicates the instructions from the tutorial above:

resource "aws_vpc_endpoint" "rds-data" {
  vpc_id              = <your-vpc-id-here>
  service_name        = "com.amazonaws.<your-region-here>.rds-data"
  vpc_endpoint_type   = "Interface"
  private_dns_enabled = true

  security_group_ids = [
    <your-security-group-ids-here>
  ]
  subnet_ids = [
    <your-subnet-ids-here>
  ]
}

AWS Lambda times out when calling RDS Serverless

2 Answers2