1

I have a lambda function that runs successfully the first time I run it (after a new deploy), but times out each subsequent time I try to run it. It definitely has something to do with RDS (MySql) because if I take the query call out, it works normally. I can console.log whatever I like out to CloudWatch, either before or after the query to RDS and all the logs make it into CloudWatch, but no errors are thrown. It's clearly executing the entire lambda, just not returning. It's also not the lambda callback code, because, again, I can simply remove the RDS query and I get a response without a hiccup.

Trevor
  • 13,085
  • 13
  • 76
  • 99

1 Answers1

0

Turns out, it's because I'm not shutting down the RDS connection. Apparently, lambda will not respond as long as an RDS connection remains open. Of course this makes sense, since you could potentially end up with an unbounded number of connections open (depending on the load of your lambda function). Unfortunately, lambda has no connection pooling feature for RDS. :( Perhaps this won't be an issue when Aurora Serverless is released?

Trevor
  • 13,085
  • 13
  • 76
  • 99
  • I'm not sure you've diagnosed this correctly. You can leave the connection open and reuse it across invocations, but you need to set the MySQL [`@@wait_timeout`](https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_wait_timeout) to a more appropriate value than the 8 hour default so that a surge of connections won't persist so long... but Lambda has no automatic protections against this particular footgun. – Michael - sqlbot Dec 06 '17 at 10:13
  • 1
    If this is Node.js, not setting `context.callbackWaitsForEmptyEventLoop = false` will cause Lambda to time out, because the event loop is held up indefinitely by that open connection... but this would impact all invocations, including the first one: https://stackoverflow.com/a/46062394/1695906 – Michael - sqlbot Dec 06 '17 at 10:16
  • `callbackWaitsForEmptyEventLoop` is most likely what I'm missing. I wasn't aware of that. Thanks! – Trevor Dec 07 '17 at 22:04