3

I have an API running in AWS Lambda and AWS Gateway using Up. My API creates a database connection on startup, and therefore Lambda does this when the function is triggered for the first time. My API is written in node using Express and pg-promise to connect to and query the database.

The problem is that Lambda creates new instances of the function as it sees fit, and sometimes it appears as though there are multiple instances of it at one time.

I keep running out of DB connections as my Lambda function is using up too many database handles. If I log into Postgres and look at the pg_stat_activity table I can see lots of connections to the database.

What is the recommended pattern for solving this issue? Can one limit the number of simultaneous instances of a function in Lambda? Can you share a connection pool across instances of a function (I doubt it).

UPDATE

AWS now provides a product called RDS Proxy which is a managed connection pooling solution to solve this very issue: https://aws.amazon.com/blogs/compute/using-amazon-rds-proxy-with-aws-lambda/

cubabit
  • 2,527
  • 3
  • 21
  • 34
  • You might be able to do something with [PgBouncer](https://wiki.postgresql.org/wiki/PgBouncer). – IMSoP Mar 08 '18 at 16:27

2 Answers2

3

There a couple ways that you can run out of database connections:

  1. You have more concurrent Lambda executions than you have available database connections. This is certainly possible.
  2. Your Lambda function is opening database connections but not closing them. This is a likely culprit, since web frameworks tend to keep database connections open across requests (which is more efficient), but on Lambda have no opportunity to close them since AWS will silently terminate the instance.

You can solve 1 by controlling the number of available connections on the database server (the max_connections setting on PostgreSQL) and the maximum number of concurrent Lambda function invocations (as documented here). Of course, that just trades one problem for another, since Lambda will return 429 errors when it hits the limit.

Addressing 2 is more tricky. The traditional and right way of dealing with database connection exhaustion is to use connection pooling. But with Lambda you can't do that on the client, and with RDS you don't have the option to do that on the server. You could set up an intermediary persistent connection pooler, but that makes for a more complicated setup.

In the absence of pooling, one option is to create and destroy a database connection on each function invocation. Unfortunately that will add quite a bit of overhead and latency to your requests.

Another option is to carefully control your client-side and server-side connection parameters. The idea is first to have the database close connections after a relatively short idle time (on PostgreSQL this is controlled by the tcp_keepalives_* settings). Then, to make sure that the client never tries to use a closed connection, you set a connection timeout on the client (how to do so will be framework dependent) that is shorter than that value.

My hope is that AWS will give us a solution for this at some point (such as server-side RDS connection pooling). You can see various proposed solutions in this AWS forum thread.

Kevin Christopher Henry
  • 46,175
  • 7
  • 116
  • 102
  • The problem I see with option 1 is that my lambda function should be able to handle a greater number of concurrent invocations than there are db connections as I am using node and pg-promise in particular. pg-promise manages a connection pool and node is asynchronous by nature. The issue is when AWS spawns too many instances of the function and the issue occurs when **instances x connection pool size > pg max connections**. i.e. I need to limit the number of consecutive **instances** of the function, not the number of invocations. – cubabit Mar 12 '18 at 09:28
  • @cubabit: We're using terminology differently. The number of concurrent instances is exactly what I'm talking about, and is what's addressed by the documentation I linked to ("In Reserve concurrency, set the value to the maximum of concurrent executions you want reserved for the function...") – Kevin Christopher Henry Mar 12 '18 at 09:54
  • Thanks, yes I read that article and had seen it before. It does say 'You can optionally set the concurrent execution limit for a function' I do not want to limit the number of concurrent executions, as I do not want each lambda function to handle just one invocation at a time - node can handle many simultaneous invocations and currently does in our environment. I just want to limit the number of simultaneous instances of the function – cubabit Mar 12 '18 at 10:01
  • @cubabit: "I do not want each lambda function to handle just one invocation at a time". Then I have some bad news - that is simply how Lambda works. The whole point is for the scaling to happen at the Lambda function level. The fact that, internally, your function uses an asynchronous platform is irrelevant. See [this question](https://stackoverflow.com/questions/45780776/how-does-aws-lambda-serve-multiple-requests) for a fuller explanation. – Kevin Christopher Henry Mar 12 '18 at 10:17
  • Yes, sure, I thought so. It is not just that I use an async implementation, but also that Lambda keeps the function 'alive' for a period of time after invocation that is enabling my app to work OK most of the time. e.g. the connections are pooled in the function, and the function stays alive for long enough to handle a number of subsequent requests. I think limiting the number of invocations so drastically is not realistic, therefore I will explore the external connection pool option. Thanks! – cubabit Mar 12 '18 at 10:26
  • @kevin-christoper-henry: Just to let you know I used pgbouncer (behind a load balancer) to pool my connections and it seems to be working well. Thanks for the advice – cubabit Mar 14 '18 at 15:23
0

You have two options to fix this:

  1. You can tweak Postgres to disconnect those idle connections. This is the best way but may require some trial-and-error.

  2. You have to make sure that you connect to the database inside your handler and disconnect before your function returns or exits. In express, you'll have to connect/disconnect while inside your route handlers.

Noel Llevares
  • 15,018
  • 3
  • 57
  • 81