1

The Postgres query builder my Lambda functions use, Knex, uses prepared statements so I'm unable to fully take advantage of RDS Proxy since the sessions are pinned. I try to ensure that the lambdas run for as little time as possible so that the pinned session completes as quickly as possible and its connection is returned to the pool.

I was wondering how I might be able to make the sessions shorter and more granular and thinking about creating and closing a connection to AWS RDS Proxy with each query.

What performance considerations should I be considering to determine the viability of this approach?

Things I'm thinking of:

  • RDS Proxy connection overhead (latency and memory)
  • The time that RDS Proxy takes to return a closed connection back to the pool and make it reusable by others (haven't been able to find documentation on this)
  • Overhead of Knex's local connection pool
Eugene Kim
  • 496
  • 4
  • 17

1 Answers1

2

Using RDS proxy when building applications with Lambda functions is a recommended infrastructure pattern by AWS. Relational Databases are not built to handle tons of connections, while Lambdas can scale to thousands of instances.

RDS Proxy connection overhead (latency and memory)

This would definitely increase your latency, but you will see a great improvement in the CPU and memory usage of your database, which would ultimately prevent unnecessary failures. It's a good trade-off when you can do a lot of other optimizations on the lambda side.

The time that RDS Proxy takes to return a closed connection back to the pool and make it reusable by others (haven't been able to find documentation on this)

While working with Lambdas, you should drop the connection to your RDS proxy, as soon as you finish processing your logic without worrying about the time the RDS proxy would take to return the closed connection back. Once the connection is dropped, the RDS proxy keeps it warm in the pool of connections it maintains for a certain duration of time. If another lambda tries to make a connection meanwhile, it can share the same connection which is still warm in the pool. Dropping the database connection at the right time from your lambda would save you lambda processing time -> money.

Overhead of Knex's local connection pool

I would suggest not using Knex local connection pool with lambda as it won't do any good (Keep the pool max to 1). Every lambda execution is independent of another, the pool will never be shared and the connection doesn't persist after the execution completes unless you plan to use it with a serverless-offline kind of local framework for development purposes.

Read More about AWS Lambda + RDS Proxy usage: https://aws.amazon.com/blogs/compute/using-amazon-rds-proxy-with-aws-lambda/

AWS Documentation on where to use RDS Proxy: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/rds-proxy-planning.html

Rajan Sharma
  • 2,211
  • 3
  • 21
  • 33
  • "...without worrying about the time the RDS proxy would take to return the closed connection back." Could you elaborate on why I shouldn't worry about this? I'm following everything you've said in your response but this is the key part on whether or not this approach is viable. My application doesn't require low latency, but if too many DB connections are in this transitory state and don't become available quickly enough, it would be problematic. – Eugene Kim Oct 09 '22 at 19:12
  • "I would suggest not using Knex local connection pool with lambda as it won't do any good." I wouldn't be using the connection pool in a traditional way but configuring the pool is required AFAIK. Typically I've seen lambda users who work with Knex set the connection pool max to 1 – Eugene Kim Oct 09 '22 at 19:13
  • The whole purpose of using RDS Proxy is to not open/close Database connections unnecessarily, that is exactly what RDS proxy does, when the lambda execution completes, you close the DB connection, but RDS proxy keeps it warm in its pool, so it can be utilised again by another lambda invocation. That also means, under the hood RDS Proxy, doesn't open a new connection to the DB unless all the connections in the pool die, which saves you DB resources. – Rajan Sharma Oct 10 '22 at 13:18
  • "Typically I've seen lambda users who work with Knex set the connection pool max to 1" - This is exactly the same what i am referring to, keep the pool max to 1, Updated the answer. – Rajan Sharma Oct 10 '22 at 13:19
  • 1
    To test the efficiency of the RDS proxy, you can actually do a load test. Trigger your Lambda function x number of times, and check the active connections for the RDS instance in cloud watch. This would give you a clear picture of how and to what extent the connection handoff is happening. – Rajan Sharma Oct 10 '22 at 13:31