0

I am using Mongo java driver 3.11.1 and Mongo Version 4.2.0 for my development.I am still learning mongo. My application receives data and either have to do insert or replace the existing document i.e. do an upsert.

Each document size is 780-1000 bytes as of now and each collection can have more than 3 millions records. Approach 1: I tried using findOneandreplace for each document and it was taking more than 15 minutes to save the data. Approach-2 I changed it to bulkwrite using below, which resulted in ~6-7 minutes for saving 20000 records.

List<Data> dataList;
dataList.forEach(data-> {
    Document updatedDocument = new Document(data.getFields());
    updates.add(new ReplaceOneModel(eq("DataId", data.getId()), updatedDocument, updateOptions));

  });
final BulkWriteResult bulkWriteResult = mongoCollection.bulkWrite(updates);

3) I tried using collection.insertMany which takes 2 seconds to store the data. As per driver code, insertMany also Internally InsertMany uses MixedBulkWriteOperation for inserting the data similar to bulkWrite.

My queries are -

a) I have to do upsert operation, Please let me know where i am doing any mistakes. - Created the indexes on DataId field but resulted in <2 miliiseconds difference in terms of performance. - Tried using writeConcern of W1, but performance is still the same.

b) why insertMany's performance is faster than bulk write. I could understand in terms of few seconds difference but unable to figure out the reason for 2-3 seconds for insertMany and 5-7 minutes for bulkwrite.

c) Are there any approaches that can be used to solve this situation.

Simulant
  • 19,190
  • 8
  • 63
  • 98
RGoyal
  • 165
  • 3
  • 16

1 Answers1

0

This problem was solved to greater extent by adding index on DataId Field. Previously i had created index on DataId field but forgot to create index after creating collection. This link How to improve MongoDB insert performance helped in resolving the problem

RGoyal
  • 165
  • 3
  • 16