Removing duplicates in MongoDB

Question

Hi everyone i've been trying for days now to remove from my db all the duplicates that are in it. I've tried this solution that I found at this Link or at this one which is almost the same but unfortunately it returns the error that forEach is not a function. I don't understand why for them works and not for me even if the code is almost the same. Here's the code that i've tried so far

exports = function(payload, response) {
  const mongodb = context.services.get("mongodb-atlas");
  var obj=EJSON.parse(payload.body.text())
  var inserimentoDB = mongodb.db("test").collection("test0").insertMany(obj)
  var duplicates = [];
  mongodb.db("test").collection("test0").aggregate([
    { "$group": {
        "_id": { "Val": "$Val" },
        "dups": { "$push": "$_id" },
        "count": { "$sum": 1 }
    }},
    { "$match": { "count": { "$gt": 1 } }}
]).toArray().forEach(function(doc) {
    doc.dups.shift();
    mongodb.db("test").collection("test0").remove({ "_id": {"$in": doc.dups }});
});
  }

I'm executing this code in a stitch function

How do you define "duplicate" and what are you trying to accomplish? — D. SM, Jun 15 '20 at 17:13
I mean that there are more than one elements in db with the same value on a attribute — phil_j_fry, Jun 16 '20 at 13:30

Bruce Bruce · Answer 1 · 2022-07-13T06:01:03.107

Not sure if this is still an issue for you, but I had a similar issue. Using MongoDB Atlas service, what worked for me was to first test out the aggregation in the aggregation widget to make sure I was getting the expected results. Then I had reformat your code a bit and use .then() instead of .forEach().

so for your case, the final solution would look something like this:

collection.aggregate([
    { "$group": {
        "_id": { "Val": "$Val" },
        "dups": { "$push": "$_id" },
        "count": { "$sum": 1 }
    }},
    { "$match": { "count": { "$gt": 1 } }}
]).toArray().then(data => {
    var dr = data.map(d => d.name);
    console.log("duplicate Records:: ", data);
    
    collection.remove({ id: { $in: dr } }).then(removedD => {
        console.log("Removed duplicate Data:: ", removedD);
    })
});

Removing duplicates in MongoDB

1 Answers1