How do i remove duplicates in mongodb?

Question

I have a database which consists of few collections , i have tried copying from one collection to another . In this process connection was lost and had to recopy them now i find around 40000 records duplicates.

Format of my data:

{ 
    "_id" : ObjectId("555abaf625149715842e6788"), 
    "reviewer_name" : "Sudarshan A", 
    "emp_name" : "Wilson Erica", 
    "evaluation_id" : NumberInt(550056), 
    "teamleader_id" : NumberInt(17199), 
    "reviewer_id" : NumberInt(1659), 
    "team_manager" : "Las Vegas", 
    "teammanager_id" : NumberInt(12245), 
    "team_leader" : "Thomas Donald", 
    "emp_id" : NumberInt(7781)
}

here only evaluation id is unique.

Queries that i have tried:

ensureIndex({id:1}, {unique:true, dropDups:true})

`ensureIndex({_id:1}, {unique:true, dropDups:true})`. Did you forget the underscore? — ZeMoon, May 20 '15 at 07:39
If `emp_id` is unique then try this `ensureIndex({"emp_id":1}, {"unique":true, "dropDups":true})` — Neo-coder, May 20 '15 at 07:42
Which indexes have you got in the table? Which version of Mongo have you got? — Alberto J García Gago, May 20 '15 at 07:43
According to my knowledge _id will no have duplicate what ever be the case — The6thSense, May 20 '15 at 08:06
usually I would go with `db.coll.find().forEach(function(doc) { db.coll2.save(doc); });` then there will not be any duplicates even if you have to recopy them. So how did you do it exactly? — yaoxing, May 20 '15 at 08:15
i have 3 lakhs of records in one collection and 1.5 lacs in other ,while moving them connection was lost i had to redo it again now i have 4.9 lacs instead of 4.5 lacs .Ineed to get rid of those 49 thousand records — ravi, May 21 '15 at 07:45
Is there a allternate for dropDups after ensuring index have also tried sarsing but no luck — ravi, May 21 '15 at 07:46

score 0 · Answer 1 · edited May 23 '17 at 10:27

0

dropDups was removed in mongodb ~2.7.

Here is other realization method

but I don't test it

edited May 23 '17 at 10:27

Community

1
1

answered Nov 04 '15 at 02:52

FuWensong

1
2

“While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes.” – Kevin Nov 04 '15 at 03:11

How do i remove duplicates in mongodb?

1 Answers1