0

I was testing the bulk update on MongoDB through node.js API. For updating 50000 entries, it takes around 8 minutes 30 seconds and then MongoDB finishes its job but not getting the API response from Node.js API.

In this process CPU usage is 32% & disk usage is 100% I mentioned the basic code here by which you guys get some idea:

// I am using mongoose.
let filteredArray = [...]//contains 50000 entries
console.time()


let bulkOps = [];
filteredArray.map( (x) => {
    let upsertDoc = {
        'updateOne': {
            'filter': { keywords: x.keywords },
            'update': {
                $inc: { counter: -1 },
                $pull: {
                    data: {
                        dataid: req.body.dataid
                    },
                },
            },
            'upsert': true
        }
    }
bulkOps.push(upsertDoc);
});
await TestDataTable.bulkWrite(bulkOps);
console.timeEnd()
return res.send({status: 200, message: "Enteries updated successfully"})

Can someone have any idea why node.js not responding and return the response? Any help is really appreciated on that.

Another approach This approach takes 2 minute 10 seconds around to complete the task.

In this process CPU usage is 100% & disk usage is 100%

console.time()
                    await Promise.all(
                        filteredArray.map(async (x) => {
                            await TestDataTable.updateOne(
                                { keywords: x.keywords },
                                {
                                    $inc: { counter: -1 },
                                    $pull: {
                                       data: {
                                          dataid: req.body.dataid
                                       },
                                    },
                                },
                            ).exec();
                        }),
                    );
                    console.timeEnd()
return res.send({status: 200, message: "Enteries updated successfully"})

But here also not getting the API response.

Aks
  • 1,092
  • 3
  • 12
  • 29
  • does bulkWrite returns a promise or only works with a callback? (and btw, independantly in the code you wrote, you do not handle the case of a failure to send a 500)) – grodzi Dec 05 '19 at 11:57
  • I have that scenario, but I not mentioned because I think that not need here. – Aks Dec 05 '19 at 12:10
  • Are you sure there is no socket timeout? 8 mins sounds quite bad for a single http request. Could you elaborate "not getting the API response". What tcpdump says about it? – Alex Blex Dec 05 '19 at 12:20
  • So after completion of the MongoDB update task, I expect the response from Node.js that Entries are updated. But it is completely blank, means no output response. And yes it takes around 8 minute. If I used loop and under that one by one I update the entries then it takes me around 2 minute 30 second. I tested that one also – Aks Dec 05 '19 at 12:26
  • @AlexBlex I updated the question. Please have a look at that – Aks Dec 05 '19 at 12:31
  • Do you have any HTTP headers in the response? What was the last tcp packets from the server? – Alex Blex Dec 05 '19 at 12:43
  • On headers I am passing the [{"key":"Content-Type","value":"application/json"}]. I don't get about tcp packets. I am on windows 10 platform – Aks Dec 05 '19 at 12:45
  • I am asking about headers that server returns. You said the problem is the **response** "is completely blank," - I am asking how "blank" it is. It is clear you have no response body, but there might be response headers, response code. If not on HTTP level, there must be something on TCP level. Otherwise the connection remains open and you are still waiting for the response. Check what's going on on transport level with tcpdump or wireshark. Any sniffer in fact. Or at least try curl request with `-v` option to get response headers. – Alex Blex Dec 05 '19 at 13:23
  • follow the steps 1. set timeout to http to wait in client side. 2. Use settimeout(0) on api function to wait for the code execution. 3. Try to chunk or split the bulk array to 10000 count anddo bulkwrite. because node js heap size will not allow to add 50000 items to array. – PrasathBV Dec 06 '19 at 07:55

0 Answers0