3

Env.:

  • AWS Lambda (Node.js, v. 8.10), waitForEmptyEventLoop === false
  • MongoDB (Atlas)
  • Mongoose

Problem: sometimes (randomly) I get next error: 

MongoNetworkError: connection 6 to db_host:27017 timed out
  File "/opt/nodejs/node_modules/mongodb-core/lib/connection/connection.js", line 259, col 7, in TLSSocket.<anonymous>
    new MongoNetworkError(f('connection %s to %s:%s timed out', self.id, self.host, self.port)),
  File "events.js", line 313, col 30, in Object.onceWrapper
  File "events.js", line 106, col 13, in emitNone
  File "events.js", line 208, col 7, in TLSSocket.emit
  File "net.js", line 420, col 8, in TLSSocket.Socket._onTimeout
  File "timers.js", line 482, col 11, in ontimeout
  File "timers.js", line 317, col 5, in tryOnTimeout
  File "timers.js", line 277, col 5, in Timer.listOnTimeout

Code of db connection:

const mongoose = require('mongoose');
const log = require('./log');

const options = {
  reconnectTries: 30,
  reconnectInterval: 500,
  poolSize: Number(process.env.DB_POOLSIZE) || 1,
  socketTimeoutMS: 30000,
  keepAlive: true,
  bufferCommands: false,
  bufferMaxEntries: 0,
};

let isConnected;

module.exports.connect = () => new Promise((resolve, reject) => {
  if (isConnected) {
    return resolve();
  }

  return mongoose.connect(process.env.DB_URI, options)
    .then((db) => {
      isConnected = db.connections[0].readyState;
      resolve();
    }).catch((error) => {
      log.error('DB:', error);
      reject(error);
    });
});

I've checked - isConnected cached successfully, mongodb connection cached in mongoose. All must be okay, but sometimes I get this error.

Anybody have any ideas how can I solve this issue?

Max Vinogradov
  • 1,343
  • 1
  • 13
  • 31
  • Same problem here, in production/dev environments I see this error on about 1% of executions. Did you find any solution/workaround? – Rodrigo Reis Jun 03 '19 at 18:10
  • @RodrigoReis I use next config (mongoose): ```{ reconnectTries: 30, reconnectInterval: 500, poolSize: 1, socketTimeoutMS: 2000000, keepAlive: true, }``` – Max Vinogradov Jun 03 '19 at 19:31
  • Just increase socketTimeoutMS - for me 2000000 enough to keep connection between lambdas invocation of "warm" container – Max Vinogradov Jun 03 '19 at 19:33
  • Another variant (smbd treat this as best practice) - create/close connection for each lambda invocation. Not bad idea if you know that lambda will invoked rarely – Max Vinogradov Jun 03 '19 at 19:35
  • Thank you @max-vinogradov . I'll try the mentioned configurations. If it really works for you you should post as an Answer :) – Rodrigo Reis Jun 03 '19 at 20:22
  • @RodrigoReis Please, notify here, and if all is okay, - I will create answer. In any way - if you know, that you usually have "cold starts", better to make new connection for each Lambda invocation – Max Vinogradov Jun 04 '19 at 06:17
  • running for 24+ hours, so far so good.. Creating the connection for each new lambda invocation will penalize too much my performance. This approach will work better I think. – Rodrigo Reis Jun 05 '19 at 01:32

2 Answers2

3

Just increase socketTimeoutMS - for me 2000000 enough to keep connection between lambdas invocation of "warm" container. Use next config (mongoose):

 { reconnectTries: 30, reconnectInterval: 500, poolSize: 1, socketTimeoutMS: 2000000, keepAlive: true, }

Another variant (smbd treat this as best practice) - create/close connection for each lambda invocation. Not bad idea if you know that lambda will invoked rarely

Max Vinogradov
  • 1,343
  • 1
  • 13
  • 31
2

You need a 'keep warm' script/functionality that will keep the connection alive/warm by querying every so often.

This issue is caused by DB connections timing out or expiring since Lambda spins up a new instance of computing every time the function is executed. It doesn't keep the existing connection warm.

This is a known issue with serverless whereby DB connection persistence is intermittent.

electric.Kariim
  • 110
  • 7