How do I use insertMany with Mongoose when I have a big array of object?

Question

I'm trying to use the insertMany method, but without success... I read this stackoverflow answer, but it does not tell how to do handle array of objects that are very big.

In my test case I have 9755 objects, with this script it imports all lines 9 times, which is not what I want...

I would like to dive the array in chunks of 1000 objects for each import.

How can I do this?

function bulkImportToMongo(arrayToImport, mongooseModel) {
  const Model = require(`../../../models/${mongooseModel}`);
  let counter = 0;
  let temp = [];
  arrayToImport.forEach(item => {
    temp.push(item);
    counter++;

    if (counter % 1000 == 0) {
      Model.insertMany(temp).then(mongoosedocs => {
        console.log(`imported ${counter} objects`);
        console.log(mongoosedocs.length);
        temp = [];
      });
    }
  });
}

score 1 · Accepted Answer · edited Sep 27 '17 at 11:27

1

You need to create an array of batches (each batch with no more than 100 elements). For each batch call Model.insertMany. To wait until all documents have been inserted, you Promise.all:

function bulkImportToMongo(arrayToImport, mongooseModel) {
  const Model = require(`../../../models/${mongooseModel}`);
  const batchSize = 100;
  let batchCount = Math.ceil(arrayToImport.length / batchSize);
  let recordsLeft = arrayToImport.length;
  let ops = [];
  let counter = 0;
  for (let i = 0; i < batchCount; i++) {
    let batch = arrayToImport.slice(counter, counter + batchSize);
    counter += batchSize;
    ops.push(Model.insertMany(batch));
  }
  return Promise.all(ops);
}

edited Sep 27 '17 at 11:27

Isak La Fleur

4,428
7
34
50

answered Sep 26 '17 at 10:52

alexmac

19,087
7
58
69

Unhandled rejection SyntaxError: Identifier 'batch' has already been declared. Should i remove "let batch ="? – Isak La Fleur Sep 26 '17 at 11:22
No, I've fixed the example. – alexmac Sep 26 '17 at 11:25
Thank you! Still not working for me. :-(. Here is the complete code if you have time to have a look.. https://repl.it/LdUd/5 – Isak La Fleur Sep 26 '17 at 11:33
Unfortunately I can't fix all bugs in your code, you asked the question about the concrete problem, I've provided the answer. – alexmac Sep 26 '17 at 11:38
Found one issue in your code example, fix `batchSize` in `Math.ceil`, instead of 1000, use 100. – alexmac Sep 26 '17 at 12:19
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/155333/discussion-between-isak-la-fleur-and-alexmac). – Isak La Fleur Sep 26 '17 at 12:24

How do I use insertMany with Mongoose when I have a big array of object?

1 Answers1