Easiest way to load documents from MongoDB collection in chunks of 10

Question

I want to create an application that creates orders and then loads them on a list. The problem is that I cannot load all orders at once. I want to load them 10 by 10.

However, this looks very difficult to do in MongoDB, as there are no automatic id like in SQL. I know how to simulate an automatic id, but I think it should be easier to implement this feature.

So, given a MongoDB collection, how do I load documents 10 by 10, starting from the latest one up to the beggining?

You could probably use a [`Cursor`](https://mongodb.github.io/node-mongodb-native/1.4/api-generated/cursor.html) with a limit and a sort. You can find some examples [here](https://stackoverflow.com/questions/25507866/how-can-i-use-a-cursor-foreach-in-mongodb-using-node-js). — cbr, Feb 03 '20 at 23:19
@cubrr I think I forgot to specify that I access things from a REST API, so I cannot use a cursor because it requires state to be saved — Guerlando OCs, Feb 03 '20 at 23:49
Open a cursor with a batchSize of 10 with whatever sort option you need, each time you process 10 documents, the driver should automatically fetch the next batch of 10 — Joe, Feb 04 '20 at 04:12
@Joe but this way the API is not REST because I must maintain a state. That is, the cursor needs to be kept on the server. — Guerlando OCs, Feb 04 '20 at 17:49

SuleymanSah · Accepted Answer · 2020-02-04T11:54:50.370

You need to sort your document by a field then use skip & limit aggregation. Also to get the total number of records we can use facet aggregation.

Here is a detailed explanation:

Let's say you have these 8 documents in orders collection.

[
    {
        "_id": "5e390fc33285e463a0799689",
        "customer": "Max",
        "total": 10,
        "orderDate": "2020-02-04T06:31:31.311Z",
        "__v": 0
    },
    {
        "_id": "5e390fd03285e463a079968a",
        "customer": "John",
        "total": 9.2,
        "orderDate": "2020-02-04T06:31:44.190Z",
        "__v": 0
    },
    {
        "_id": "5e390fda3285e463a079968b",
        "customer": "Smith",
        "total": 11.3,
        "orderDate": "2020-02-04T06:31:54.248Z",
        "__v": 0
    },
    {
        "_id": "5e390fdf3285e463a079968c",
        "customer": "Smith",
        "total": 12.3,
        "orderDate": "2020-02-04T06:31:59.993Z",
        "__v": 0
    },
    {
        "_id": "5e390fec3285e463a079968d",
        "customer": "Jimmy",
        "total": 15.6,
        "orderDate": "2020-02-04T06:32:12.336Z",
        "__v": 0
    },
    {
        "_id": "5e390ffd3285e463a079968e",
        "customer": "Wesley",
        "total": 11,
        "orderDate": "2020-02-04T06:32:29.670Z",
        "__v": 0
    },
    {
        "_id": "5e3910163285e463a079968f",
        "customer": "Adam",
        "total": 6.1,
        "orderDate": "2020-02-04T06:32:54.131Z",
        "__v": 0
    },
    {
        "_id": "5e3910213285e463a0799690",
        "customer": "Michael",
        "total": 7.2,
        "orderDate": "2020-02-04T06:33:05.166Z",
        "__v": 0
    }
]

If we wanted to get these documents in chunks, we can write a sample route like this:

router.get("/orders", async (req, res) => {
  const page = req.query.pageIndex ? +req.query.pageIndex : 1;
  const limit = req.query.pageSize ? +req.query.pageSize : 10;
  const skip = (page - 1) * limit;

  const result = await Order.aggregate([
    {
      $sort: {
        orderDate: -1
      }
    },
    {
      $facet: {
        totalRecords: [{ $count: "total" }],
        data: [{ $skip: skip }, { $limit: limit }]
      }
    }
  ]);
  res.send(result);
});

We send the pageIndex and pageSize parameters in query string like this http://...../orders?pageIndex=1&pageSize=3

When we use pageIndex=1 and pageSize=3, the result will be like this: (as you see we also return the total number of records so that client can build the pagination numbers)

[
    {
        "totalRecords": [
            {
                "total": 8
            }
        ],
        "data": [
            {
                "_id": "5e3910213285e463a0799690",
                "customer": "Michael",
                "total": 7.2,
                "orderDate": "2020-02-04T06:33:05.166Z",
                "__v": 0
            },
            {
                "_id": "5e3910163285e463a079968f",
                "customer": "Adam",
                "total": 6.1,
                "orderDate": "2020-02-04T06:32:54.131Z",
                "__v": 0
            },
            {
                "_id": "5e390ffd3285e463a079968e",
                "customer": "Wesley",
                "total": 11,
                "orderDate": "2020-02-04T06:32:29.670Z",
                "__v": 0
            }
        ]
    }
]

When we use pageIndex=2 and pageSize=3, the result will be like this:

[
    {
        "totalRecords": [
            {
                "total": 8
            }
        ],
        "data": [
            {
                "_id": "5e390fec3285e463a079968d",
                "customer": "Jimmy",
                "total": 15.6,
                "orderDate": "2020-02-04T06:32:12.336Z",
                "__v": 0
            },
            {
                "_id": "5e390fdf3285e463a079968c",
                "customer": "Smith",
                "total": 12.3,
                "orderDate": "2020-02-04T06:31:59.993Z",
                "__v": 0
            },
            {
                "_id": "5e390fda3285e463a079968b",
                "customer": "Smith",
                "total": 11.3,
                "orderDate": "2020-02-04T06:31:54.248Z",
                "__v": 0
            }
        ]
    }
]

When we use pageIndex=3 and pageSize=3, the result will be like this:

[
    {
        "totalRecords": [
            {
                "total": 8
            }
        ],
        "data": [
            {
                "_id": "5e390fd03285e463a079968a",
                "customer": "John",
                "total": 9.2,
                "orderDate": "2020-02-04T06:31:44.190Z",
                "__v": 0
            },
            {
                "_id": "5e390fc33285e463a0799689",
                "customer": "Max",
                "total": 10,
                "orderDate": "2020-02-04T06:31:31.311Z",
                "__v": 0
            }
        ]
    }
]

For your case, you need to send 10 as pageSize value.

@GuerlandoOCs I am happy that you liked my answer, accepting this as answer will be enough for me, you are very kind. — SuleymanSah, Feb 04 '20 at 18:14
Just a question: in an aggregation pipeline, do thins occur atomically? In the first step of the data pipeline, it skips x and limits y. It it possible that, while it is skiping those x, a new document is added? — Guerlando OCs, Feb 04 '20 at 18:14
@GuerlandoOCs good question, honestly I don't know, but If I find the answer I will comment here. But in general this will work well. — SuleymanSah, Feb 04 '20 at 18:16
Since we can count the total number of documents, can aggregation be used to create a new document with auto increment id? Please take a look at my question: https://stackoverflow.com/questions/60064892/can-mongodb-aggregation-be-used-to-create-auto-increment-id — Guerlando OCs, Feb 04 '20 at 20:14
@GuerlandoOCs we use aggregation framework for querying, it is not used for creating a new document. — SuleymanSah, Feb 04 '20 at 20:22
I just noticed a major problem with your approach: if a new document is added while I'm scrolling to a list that load them 10 by 10, I will get repeated documents, as the newly inserted document will shift the number of documents I skip :( — Guerlando OCs, Feb 04 '20 at 20:44
@GuerlandoOCs I didn’t understand exactly how repetition happens, can you give a scenario on the documents in my answer? — SuleymanSah, Feb 04 '20 at 20:51
Since you made ithe `orderDate` descending, `pageIndex=1 and pageSize=2` will give `Michel` and `Adam`. Then someone adds an order for `Lucas`. If you do `pageIndex=2 and pageSize=2` now you'll get `Adam` and `Lucas`. See that `Adam` repeated. So if someone adds while you're scrolling, it gives repeated items — Guerlando OCs, Feb 04 '20 at 21:29
@GuerlandoOCs this scenario is not specific to this solution, it may happen in every paging solution, and actually it is not repetition since we don’t show the same record duplicated in one page. I think this is the best paging solution you can find for mongodb. — SuleymanSah, Feb 05 '20 at 04:59

Easiest way to load documents from MongoDB collection in chunks of 10

1 Answers1

Linked