1

I'm very new to systems design in general, so let me try and explain my question to the best of my ability!

I have two EC2 t2.micro instances up and running: one is housing my MongoDB, which is storing 10,000,000 primary records, and the other has my express server on it.

The structure of my MongoDB documents are as follows:

{
  _id: 1,
  images: ["url_1.jpg", "url_2.jpg", "url_3.jpg"],
}

This is what my mongo connection looks like:

const { MongoClient } = require('mongodb');
const { username, password, ip } = require('./config.js');

const client = new MongoClient(`mongodb://${username}:${password}@${ip}`,
  { useUnifiedTopology: true, poolSize: 10 });

client.connect();

const Images = client.db('imagecarousel').collection('images');

module.exports = Images;

I am using loader.io to run a 1000PRS stress test to my servers GET API endpoint. The first test uses a .findOne() query, the second a .find().limit(1) query, like so:

const query = { _id: +req.params.id };
Images.findOne(query).then((data) => res.status(200).send(data))
    .catch((err) => {
      console.log(err);
      res.status(500).send(errorMessage);
    });

//////////////////////////////////////////

const query = { _id: +req.params.id };
Images.find(query).limit(1).toArray().then((data) => res.status(200).send(data[0]))
    .catch((err) => {
      console.log(err);
      res.status(500).send(errorMessage);
    });

When I looked at the results on New Relic, I was a little perplexed by what I saw: New Relic Results

After some research, I figured this has something to do with .findOne() returning a document, and .find() returning a cursor?

So my question is: How do I determine if the bottle neck is node.js or MongoDB, and do the queries I use determine that for me (in this specific case)?

Matthew Wigglesworth
  • 65
  • 5
  • What did you see in newrelic results and what perceived bottleneck are you referring to? – D. SM Sep 11 '20 at 04:23
  • Does this answer your question? [Retrieve Mongoose Query Queue Time Metrics](https://stackoverflow.com/questions/63652084/retrieve-mongoose-query-queue-time-metrics) – D. SM Sep 11 '20 at 04:26
  • I noticed that on .find(), the web transaction time for Mongo was sub 1ms, but with .findOne() it was 1300ms – Matthew Wigglesworth Sep 11 '20 at 05:28
  • check this package - https://github.com/clinicjs/node-clinic. And for mongoDB profiling - https://github.com/nearform/node-clinic-bubbleprof-demo – spyder Mar 05 '21 at 09:43

1 Answers1

3

I would suggest that you start with the mongodb console and explore your queries in detail. This way you will isolate the mongodb behavior from the driver behavior.

A good way to analyse your queries is:

If you aim at pitch-perfect database performance tuning, you need to understand every detail of the execution of your queries. It will take some time to get grip of it, but it's totally worth it!

Another detail of interest is the real-world performance monitoring and profiling in production, which reveals the true picture of the bottlenecks in your application, as opposed to the more "sterile" non-production stress-testing. Here is a good profiler, which allows you to insert custom profiling points in your app and to easily turn profiling on and off without restarting the app:

https://www.npmjs.com/package/mc-profiler

A good practice would be to first let the application run in production as a beta, inspect profiling data and optimize the slow code. Otherwise you could waste swathes of time going after some optimizations, which have little to no impact to the general app performance.

Danny Apostolov
  • 479
  • 3
  • 8