1

I am trying to insert multiple records in MongoDB, for a small number of records (less than 1000) it works fine. But with a huge data set it throws an error if there are any duplicate values.

Is there a way to continue without an error if there are duplicate records?

Code

    db.collection.insertMany(numberList, {
      ordered: false
    });]

Error

    { MongoError: E11000 duplicate key error collection: campaignDB.campaign_number index: id dup key: { : "5e673dc3a53cf3730568a19b_94715987119" }
    at Function.create (/Users/waruna/Anuja/GitBuddika/Message_hub/MHNG-Backend/campaign-services/node_modules/mongodb/lib/core/error.js:44:12)
    at toError (/Users/waruna/Anuja/GitBuddika/Message_hub/MHNG-Backend/campaign-services/node_modules/mongodb/lib/utils.js:150:22)
    at UnorderedBulkOperation.handleWriteError (/Users/waruna/Anuja/GitBuddika/Message_hub/MHNG-Backend/campaign-services/node_modules/mongodb/lib/bulk/common.js:1125:11)
    at resultHandler (/Users/waruna/Anuja/GitBuddika/Message_hub/MHNG-Backend/campaign-services/node_modules/mongodb/lib/bulk/common.js:501:23)
    at handler (/Users/waruna/Anuja/GitBuddika/Message_hub/MHNG-Backend/campaign-services/node_modules/mongodb/lib/core/sdam/topology.js:973:24)
    at wireProtocol.(anonymous function) (/Users/waruna/Anuja/GitBuddika/Message_hub/MHNG-Backend/campaign-services/node_modules/mongodb/lib/core/sdam/server.js:437:5)
    at /Users/waruna/Anuja/GitBuddika/Message_hub/MHNG-Backend/campaign-services/node_modules/mongodb/lib/core/connection/pool.js:420:18
    at process._tickCallback (internal/process/next_tick.js:61:11)
  name: 'BulkWriteError',
  driver: true,
  code: 11000,
  writeErrors:
   [ WriteError { err: [Object] },
     ... 899 more items ],
  result:
   BulkWriteResult {
     result:
      { ok: 1,
        writeErrors: [Array],
        writeConcernErrors: [],
        insertedIds: [Array],
        nInserted: 0,
        nUpserted: 0,
        nMatched: 0,
        nModified: 0,
        nRemoved: 0,
        upserted: [] } },
  [Symbol(mongoErrorContextSymbol)]: {} }
Shafeek Mohamed
  • 11
  • 3
  • yes remove duplicate names – vicky Mar 20 '20 at 10:12
  • Why did you create a unique index on the campaign number field if you don't want it to be unique? – Anže Mur Mar 20 '20 at 10:14
  • And how do you generate campaign numbers? You can solve this by ensuring that generated value is unique. – Anže Mur Mar 20 '20 at 10:16
  • I want only to save only the unique Numbers, but it should save the remaining numbers if there is a duplicate. Now, it throws an error without saving the numbers. – Shafeek Mohamed Mar 20 '20 at 10:16
  • The numbers are not system generated. Users upload mobile numbers through a UI. For a campaign, the mobile numbers should not be duplicated. – Shafeek Mohamed Mar 20 '20 at 10:18
  • If you want to save numbers whether they are unique or not, you should remove the unique index on the campaign number field. – Anže Mur Mar 20 '20 at 10:25
  • You may generate `.json` file with valid MongoDB structure and use `mongoimport`. It will ignore all duplicated records. Run `mongoexport` to your current collection and check how JSON looks like to replicate – Valijon Mar 20 '20 at 11:00
  • Thanks @Valijon, but I need to do this from a NodeJS application - mongodb driver, not from the Mongo CLI. – Shafeek Mohamed Mar 23 '20 at 04:28

0 Answers0