Data upsert issue using BulkWriteOperation in Java

Question

I am trying to upsert millions of data using BulkWriteOperation, but my code is giving exception when my query condition is not satisfying but a document is available with that id. Here is my code :-

if(provisionSubscriberList.size()>0){

        Map<String, Object> map = new HashMap<String, Object>();
        map.put("id", campaignTO.getId());
        map.put("testSample", false);
        map.put("status", "Active");
        map.put("controlGroup", false);
        try{
    WriteConcern wc = WriteConcern.ACKNOWLEDGED;
    BulkWriteOperation bulk = mongoTemplate.getCollection("provisionSubscriber").initializeOrderedBulkOperation();

    for (ProvisionSubscriberEntity provisionalSubscriber : provisionSubscriberList) {

        Query queryForAddSubscriber = new Query();

        Update updateFieldsForAddSubscriber = new Update();
        updateFieldsForAddSubscriber.set("msisdn", provisionalSubscriber.getMsisdn());
        updateFieldsForAddSubscriber.set("deviceType", provisionalSubscriber.getDeviceType());
        updateFieldsForAddSubscriber.addToSet("campaignIdList", map);


        List<DBObject> criteria = new ArrayList<DBObject>();
        criteria.add(new BasicDBObject("_id",new ObjectId(provisionalSubscriber.getId())));
        criteria.add(new BasicDBObject("campaignIdList.id", new BasicDBObject("$ne", campaignTO.getId())));
        criteria.add(new BasicDBObject("campaignIdList.controlGroup", new BasicDBObject("$ne", true)));
        criteria.add(new BasicDBObject("campaignIdList.status", new BasicDBObject("$ne", "Active")));
        BasicDBObject queryCriteria = new BasicDBObject("$and", criteria);

        bulk.find(queryCriteria).upsert().updateOne(updateFieldsForAddSubscriber.getUpdateObject());

    }
    BulkWriteResult results =bulk.execute(wc);
    System.out.println(results);
    for (BulkWriteUpsert up : results.getUpserts()) {
        System.out.println(up.getId());
    }

And Here is the Exception I am getting:-

com.mongodb.BulkWriteException: Bulk write operation error on server 192.168.1.113:27017. Write errors: [BulkWriteError{index=0, code=11000, message='E11000 duplicate key error index: jmailer_digiengage.provisionSubscriber.$_id_ dup key: { : ObjectId('58c8f33301de9614143f5812') }', details={ }}]. 
at com.mongodb.BulkWriteHelper.translateBulkWriteException(BulkWriteHelper.java:56)
at com.mongodb.DBCollection.executeBulkWriteOperation(DBCollection.java:2310)
at com.mongodb.BulkWriteOperation.execute(BulkWriteOperation.java:136)
at com.lumatadigital.digiengage.daoImpl.ProvisioningDaoImpl.provisionOnCampaign(ProvisioningDaoImpl.java:120)
at com.lumatadigital.digiengage.schedular.service.SchedularJobConfig.provisioningJob(SchedularJobConfig.java:29)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:269)
at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:257)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:75)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)

EDIT: Basically, I want to insert data if document is not available or update data if the document is available and my query is satisfying to that document, otherwise skip that document. Also, I want to track the upserted documents.

yes @felix that is the issue. Is there any way to do the same thing and skip the the duplicate document. — Devendra Singh, May 19 '17 at 06:09
Unordered operations don't stop of error an simply report the error. This would allow the batch to complete and simply report the errors. — Neil Lunn, May 19 '17 at 06:11

score 1 · Answer 1 · answered May 19 '17 at 06:14

It is because in your query criteria:

List<DBObject> criteria = new ArrayList<DBObject>();
criteria.add(new BasicDBObject("_id",new ObjectId(provisionalSubscriber.getId())));
criteria.add(new BasicDBObject("campaignIdList.id", new BasicDBObject("$ne", campaignTO.getId())));
criteria.add(new BasicDBObject("campaignIdList.controlGroup", new BasicDBObject("$ne", true)));
criteria.add(new BasicDBObject("campaignIdList.status", new BasicDBObject("$ne", "Active")));
BasicDBObject queryCriteria = new BasicDBObject("$and", criteria);

If the _id field is inserted in the database already with the insert statement, and when the update statement runs for the next time, the criteria "$ne" (not equal to) in the campaign list object fails that will create new row with the same _id tries to insert instead of update since the previous data do not match with the current data.

Hence you are getting the below error:

E11000 duplicate key error index: jmailer_digiengage.provisionSubscriber.$_id_ dup key: { : ObjectId('58c8f33301de9614143f5812') }

Akshay Mehta · Answer 2 · 2017-05-19T11:48:26.313

To do bulk update, you can use the below code

        MongoCollection<Document> collection = database.getCollection("collection");
        List<WriteModel<Document>> updates = new ArrayList<WriteModel<Document>>();

        UpdateOptions options = new UpdateOptions();
        options.upsert(true);
        // Doc1 update
        Document doc1 = new Document("$set", new Document("key1", "value1"));
        updates.add(new UpdateOneModel<Document>(new Document("_id",new ObjectId("562a44971bca3c0001953f42")), doc1, options));

        //Doc2 update
        Document doc2 = new Document("$set", new Document("key1", "value2"));
        updates.add(new UpdateOneModel<Document>(new Document("_id",new ObjectId("562a44971bca3c0001954071")), doc2, options));

        BulkWriteResult result = collection.bulkWrite(updates);
        System.out.println("Updated count : " + result.getModifiedCount());

In the below snippet

updates.add(new UpdateOneModel<Document>(new Document("_id",new ObjectId("562a44971bca3c0001954071")), doc2, options));

The 1st condition is the filter condition where you can use any key present in the doc to filter out the document you want to update, the 2nd parameter is the fields that needs to be updated for the doc, the 3rd parameter is additional options that can be passed to the model

This will only update the existing documents, but I want to upsert the document, insert a new document if not available. — Devendra Singh, May 19 '17 at 09:55
Ok if you want to insert a document in case it does not exist then you can pass the 3rd parameters UpdateOptions. I've just edited the code to include it — Akshay Mehta, May 19 '17 at 11:45
@Devendra Singh - Am not sure if it was helpful to you, but just incase you were able to get it to working with this method, it would be great to accept the answer and incase you had a different approach to your solution, please post the approach so that it will be helpful to someone who is looking out for a solution. — Akshay Mehta, Jul 25 '17 at 06:18

Data upsert issue using BulkWriteOperation in Java

2 Answers2