Mongodb compare two big data collection

Question

I want to compare two very big collection, the main of the operation is two know what element is change or deleted My collection 1 and 2 have a same structure and have more 3 million records example : record 1 {id:'7865456465465',name:'tototo', info:'tototo'}

So i want to know : what element is change, and what element is not present in collection 2. What is the best solution to do this?

Asya, Yes i want to know if a element is deleted or is changed. — timactive, Jan 17 '14 at 00:36
Just start looping through each, tracking success/failures. There's no better option than brute force comparisons. — WiredPrairie, Jan 17 '14 at 01:32
untrue - if there were no changes you waste a lot of time that way. — Asya Kamsky, Jan 17 '14 at 06:38
Yes Asya Kamsky, it is no possible to looping each record it is very long process. — timactive, Jan 17 '14 at 10:43

score 0 · Answer 1 · answered Jan 17 '14 at 02:33

1) Define what equality of 2 documents means. For me it would be: both documents should contain all fields with exact same values given their ids are unique. Note that mongo does not guarantee field order, and if you update a field it might move to the end of the document which is fine.

2) I would use some framework that can connect to mongo and fetch data at the same time converting it to a map-like data structure or even JSON. For instance I would go with Scala + Lift record (db.coll.findAll()) + Lift JSON. Lift JSON library has Diff function that will give you a diff of 2 JSON docs.

3) Finally I would sort both collections by ids, open db cursors, iterate and compare.

score 0 · Answer 2 · answered May 10 '18 at 18:18

0

if the schema is flat in your case it is, you can use a free tool to compare the data(dataq.io) in two tables.

Disclaimer : I am the founder of this product.

answered May 10 '18 at 18:18

firemonkey

329
4
18

Mongodb compare two big data collection

2 Answers2