0

Introduction

I am hosting a web server implemented by gorilla/mux which expose API endpoints for users to access the database.

I put the server in a AWS Lambda function and use API Gateway to send Lambda proxy requests to the server. The request flow is:

  • API Gateway -> (API Gateway Lambda proxy integration) -> Lambda function -> RDS proxy -> RDS instance

The server code

import (
    "github.com/gorilla/mux"
    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
    "github.com/awslabs/aws-lambda-go-api-proxy/core"
    "github.com/awslabs/aws-lambda-go-api-proxy/gorillamux"
)

var (
    muxLambda *gorillamux.GorillaMuxAdapter
    router = mux.NewRouter().StrictSlash(true)
    // Register handler functions
)

func init() {
    muxLambda = gorillamux.New(router)
}

func LambdaHandler(
    ctx context.Context,
    req events.APIGatewayProxyRequest,
) (*core.SwitchableAPIGatewayResponse, error) {
    response, err := muxLambda.ProxyWithContext(ctx, *core.NewSwitchableAPIGatewayRequestV1(&req))
    if err != nil {
        panic(fmt.Sprintf("Proxy encountered an error: %s", err.Error()))
    }
    return response, nil
}

func main() {
    lambda.Start(LambdaHandler)
}
  • I open the connection to database via:
import "database/sql"
database, err := sql.Open("postgres", connStr)

Problem

While testing the load performance of this server (siege -c 250 -d 0.5 -r 10 'http://<api url>/), I encountered so many connections to the RDS instance (around 400 connections in the AWS console) and the Lambda function returned timeout (30 sec) for almost all transactions. If I lower the number of process, the response from the server works as expected.

What are the possible things that cause this?

From my search, I found that maybe it is due to:

  • For each request, a new Lambda instance starts and opens a new database connection to the RDS proxy. With too many connections already there, somehow the Lambda waits and return timeout after 30 seconds.

This might be similar to this question, but I think RDS proxy can handle connection pool for me? Or am I overloading the RDS proxy?

If the cause is overloading, what configuration can I do in the AWS setup? Scaling up the RDS instances? Or Increase max_connections setting as mentioned in this article?

Or simply change the code structure as mentioned in this SO question or this SO question?

Thanks for reading this far. Any feedback is appreciated.

Updated: The connection metrics of RDS and RDS proxy

hcchang84
  • 3
  • 5
  • You should look at the connection metrics of both the RDS instance and RDS proxy to see where the issue is. – Mark B Jul 28 '22 at 13:08
  • @MarkB I have added some connection metrics, it seems the connection spiked during the load testing. – hcchang84 Jul 29 '22 at 05:46

2 Answers2

0

Increase value of max_connections is not best practice. I think, the better way to use Lambda function init phase to pre-configure RDS connection. This connection will be reused by several lambda functions. This solution decreases number of connections to RDS database. There are good example how to do this.

Maksim Yersh
  • 51
  • 1
  • This shouldn't be necessary while using RDS Proxy. – Mark B Jul 28 '22 at 13:05
  • Thanks for the advice. I was hoping RDS proxy can handle the connections for me. Do you suggest using configuration like `connect_timeout=5` from the example to limit the RDS connection time? – hcchang84 Aug 01 '22 at 06:58
0

I found the reason that Lambda timeout might due to the fact of too many RDS connections. So the RDS proxy queues or throttles incoming connections from the Lambda (ref).

The root cause was because of too many SQL queries from the API server sending to the RDS, which holds up the connections. In my case, the query includes 2 steps:

  1. Getting a list of id
  2. Sends a query for each id such as:
select row from table where id = 1;

In this setting, if there are 200 id in the list, then the API server will send 200 + 1 queries in total, which holds up the connection.

Solution

  • I modified the API server to send a single query like:
select row from table where id in (1, 2, 3, ....);
  • Close the rows from Db.Query (from database/sql) calls to avoid connection leak:
rows, err := Db.Query(query, params)
defer rows.Close()

In summary, this was due to poor understanding of the database/sql go package and SQL. Hopefully this helps those who comes across this.

hcchang84
  • 3
  • 5