How to insert multiple items items at the same time into a MongoDB collection?

Question

I have a collection of 12K models and I want to insert in my Mongo DB.

Mongo db version: 4.0.3 Mongoose version: 5.3.4

I tried with InsertMany, create, save() into forEach but I can't insert my 12K models.

const ProductSchema = new mongoose.Schema({
  name: {
    type: String,
    required: true
  },
  id: {
    type: String,
    required: true
  },
  sku: {
    type: String,
    required: true
  },
  categoriesIds: [
    {
      type: String
    }
  ]},
  {
    usePushEach: true, timestamps: true
  });

 const prodModel = new ProductModel({
              name: prod.name,
              id: prod.id,
              sku: prod.sku,
              categoriesIds: prod.categories
            });

I have array of 12K productModel created and then I did:


ProductModel.insertMany(products, (error) => {

});

Only when the length of the array is less than 1K works ok. I read about since Mongo 3.6 the maxWriteBatchSize is 100K. I'm using Mongo 4. I don't understand why not works with only 12K elements.

The console doesn't show errors.

Have you tried bulk insert? https://stackoverflow.com/a/25313401/10387396 — André Adriano, Nov 09 '18 at 16:29
Yes I just tried that thing. Same result. I use mongoose.connection for execute that initializeUnorderedBulkOp() .. — Gere, Nov 09 '18 at 16:51

score 1 · Answer 1 · answered Nov 10 '18 at 15:32

Finally I resolved using mongo driver for javascript and a batch size logic:

const batchSize = 1000;
        const col = db.collection('products');
        let batch = col.initializeUnorderedBulkOp();
        for (let i = 0; i <= products.length; i += 1) {
          if (products[i]) {
            batch.insert(products[i]);
            if (i % batchSize === 0) {
              batch.execute();
              batch = col.initializeUnorderedBulkOp();
            }
          }
        }
        return batch.execute();

How to insert multiple items items at the same time into a MongoDB collection?

1 Answers1