Mongoose creating too many connections in NextJS

Question

I've been using Mongoose to connect Mongodb with my nextjs application, everything was working fine in the development mode, but when I deployed the project, even a single user (me) is taking around 60 to 80 connections. And when I shared the link with 3/4 of my friends, the connection has reached 500 connections limit.

Here's my initDB file's code:

import mongoose from "mongoose";

const MONGO_URI = process.env.MONGO_URI;

if (!MONGO_URI) {
  throw new Error("Error Code: grx30xd33d");
}

let cached = global.mongoose;
if (!cached) {
  cached = global.mongoose = { conn: null, promise: null };
}

async function initDB() {
  if (cached.conn) {
    return cached.conn;
  }

  if (!cached.promise) {
    const opts = {
      bufferCommands: false,
      useNewUrlParser: true,
      useUnifiedTopology: true,
      maxPoolSize: 50,
      wtimeoutMS: 2500,
      useNewUrlParser: true,
    };

    mongoose.set("strictQuery", false);
    cached.promise = mongoose.connect(MONGO_URI, opts).then((mongoose) => {
      return mongoose;
    });
  }

  try {
    cached.conn = await cached.promise;
  } catch (e) {
    cached.promise = null;
    throw e;
  }

  return cached.conn;
}

export default initDB;

And this is my page which is using the initDB.

import initDB from "../../../helpers/initDB";
import directorySchema from "../../../models/directory/directorySchema ";

export default async function handler(req, res) {
  await initDB();

  return new Promise(async (resolve, reject) => {
    try {
      const userProf = await directorySchema
        .find()
        .limit(20)
        .sort({ _id: -1 })
        .then(async (response) => {
          if (response !== null) {
            res.json({
              success: true,
              data: response,
              message: "Data Fetched Successfully",
            });
          } else {
            return res.json({
              success: false,
              message: "Invalid data response",
            });
          }
        })
        .catch((error) => {
          console.log(error);
          res.json(error);
          res.status(405).end();
          return reject();
        });
    } catch (error) {
      console.log(error);
    }
  });
}

The response is working fine, I'm getting the required response, but the connection limit is reaching in mongodb.

Here's the screenshot of connection while writing this question.

Answer 1

The issue is not with next.js but with Vercel hosting. As OP stated the problem is not reproducible locally.

The issue is that serverless nature of Vercel is not quite compatible with thick stateful mongodb client. Mongodb native nodejs driver is designed to run permanently, manage socket connections to reuse them for multiple requests, keep track of replica set topology etc. Vercel on the other hand starts a new nodejs process on each HTTP request, which makes it impossible to reuse already opened connections.

The "answer from a reputable source" part:

The issue is being discussed in https://github.com/vercel/next.js/discussions/12229 and is still open at the time of answering the question.

The most practical recommendation is from the current maintainer of the next.js repo Lee Robinson is

You can use Mongo's Data API.

https://www.mongodb.com/docs/atlas/api/data-api-resources/

It indeed solves the problem with connection pool for the cost of:

• performance: each db request is and HTTP call which x100 times slower on simple requests.
• maintenance: it requires to keep external data API in sync with vercel app
• dependencies: you won't use native driver, and consequently you cannot use any libraries that depend on it - mongoose in particular in context of the question.

The less dramatic approach is recommended by a contributor Chris Anthony:

1. Ensure the connection is cached, and that you deal with the promised connection correctly. - see Next example [https://github.com/vercel/next.js/blob/canary/examples/with-mongodb/util/mongodb.js] (https://github.com/vercel/next.js/blob/canary/examples/with-mongodb/util/mongodb.js)

2. Ensure you have configured the client to kill idle connections. Where maxIdleTimeMS isn't supported, socketTimeoutMS will do.

3. Make sure you close all cursors. Previously , I was letting Mongo server do this.. but it was woeful at that. Now I make sure I 1st store the response , then manually close the cursor on each end point.

E.g a config with 1 connection in a pool:

    const config = {
        useNewUrlParser: true,
        useUnifiedTopology: true,
        maxPoolSize: 1,
        minPoolSize: 1,
        socketTimeoutMS: 10000,
        serverSelectionTimeoutMS: 10000,
        maxIdleTimeMS: 10000,
    };

will result with 1 connection per http request, but the driver won't be able to run requests in parallel. You will need to experiment with exact timeouts and pool size to find balance.

It's far from efficient use of the driver, but at least it will unblock use of dependant libraries like mongoose.

Closing connection is a bit tricky, if you do it before all query promises are resolved, you will get "MongoError: Topology is closed, please connect" error. It's especially true for Mongoose, as it's designed to use higher abstraction level and do not manage connections manually.