MongoDB seed script for 10million entries takes 30 minutes

Question

I have a project im working on and I have to seed a database with 10 million random rows, which i have successfully done. However it takes about 30 minutes for it to complete, which is expected, but i know it could be faster. I would like to make it run ever faster and figure out a way to make it seed 10 million random entries in under 10 minutes preferably while still using mongodb/mongoose. This is my current seed file, any tips on making it run faster? First time posting on here, just fyi. thanks!

I use 'node database/seed.js' to run this file in the terminal.

const db = require("./index.js");
const mongoose = require("mongoose");
const faker = require("faker");

const productSchema = mongoose.Schema({
  product_name: String,
  image: String,
  price: String
});

let Product = mongoose.model("Product", productSchema);

async function seed() {
  for (let i = 0; i < 10000000; i++) {
    let name = faker.commerce.productName();
    let image = faker.image.imageUrl();
    let price = faker.commerce.price();

    let item = new Product({
      product_name: `${name}`,
      image: `${image}`,
      price: `$${price}`
    });

    await item
      .save()
      .then(success => {})
      .catch(err => {});
  }
}
seed();

score 1 · Answer 1 · answered Feb 28 '19 at 10:31

1

You can create batch of may be 1 million records and can use insertMany function to insert bulk into database.

answered Feb 28 '19 at 10:31

Shriyash Warghade

183
1
4

If my memory serves me, there is a limit on the number of records for bulk API, but in general yes, this is a use case for Bulk API... Not sure what mongoose uses under the hood, though. – RidgeA Feb 28 '19 at 10:41
1

@RidgeA The [MongoDB NodeJS Driver](http://mongodb.github.io/node-mongodb-native/3.1/) of course. Check the `package.json`. There used to be a cap of 1000 documents per "batch to server" ( No actual limit on what you throw at the method itself ) but that's gone now as long as the server is 3.6 or greater. The only real cap now is 16MB on what actually goes per "batch" request. The real reason you would not do "1 million" is more about creating such are large in-memory array. And of course that a few smaller net requests are better than one fat one, or of course millions of "tiny" ones – Neil Lunn Mar 01 '19 at 11:42

score 0 · Answer 2 · answered Feb 28 '19 at 11:00

0

Use InsertMany

Insert/update always takes time in all kinda database. try to reduce number of insertion.

Insert something for every 1000 or loops once

Model.insertMany(arr, function(error, docs) {});

answered Feb 28 '19 at 11:00

Thamaraiselvam

6,961
8
45
71

MongoDB seed script for 10million entries takes 30 minutes

2 Answers2