2

I need to insert 10K documents as fast as possible but it's taking a long time.

I am currently using Model.create([<huge array here>]) to do this.

Would it help to use multiple connections to the database? For example have 10 connections saving 1K each?

1sina1
  • 937
  • 7
  • 11
emorling
  • 147
  • 1
  • 2
  • 11
  • check [this](https://stackoverflow.com/questions/10266512/how-can-i-save-multiple-documents-concurrently-in-mongoose-node-js) – 1sina1 Mar 01 '22 at 19:37

2 Answers2

4

You can use model.insertMany(doc, options).

Some stuff to note below.

Connection Pool

10 connections is usually sufficient, but it greatly depends on your hardware. Opening up more connections may slow down your server.

In some cases, the number of connections between the applications and the database can overwhelm the ability of the server to handle requests.

Options

There are a couple of options for insertMany that can speed up insertion.

[options.lean «Boolean» = false] if true, skips hydrating and validating the documents. This option is useful if you need the extra performance, but Mongoose won't validate the documents before inserting.

[options.limit «Number» = null] this limits the number of documents being processed (validation/casting) by mongoose in parallel, this does NOT send the documents in batches to MongoDB. Use this option if you're processing a large number of documents and your app is running out of memory.

Write concern

Setting options on writeConcern in options can also affect performance.

If applications specify write concerns that include the j option, mongod will decrease the duration between journal writes, which can increase the overall write load.

Someone Special
  • 12,479
  • 7
  • 45
  • 76
  • Do you have any input on the idea of splitting the 10K model.insertMany into 10 separate requests to use 10 separate connections? Would this be faster? – emorling Mar 02 '22 at 20:10
  • How big is your document? What is your use case? If you want this kind of comparison you have to run your own test. – Someone Special Mar 02 '22 at 23:40
0

Use db.collection.insertMany([])

insertMany will accept array of objects and this is use to perform bulk insertions.

Mustafa Jawed
  • 48
  • 6
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Mar 02 '22 at 05:55