ReplaceOne throws duplicate key exception

Question

My app receives data from a remote server and calls ReplaceOne to either insert new or replace existing document with a given key with Upsert = true. (the key is made anonymous with *) The code only runs in a single thread.

However, occasionally, the app crashes with the following error:

Unhandled Exception: MongoDB.Driver.MongoWriteException: A write operation resulted in an error.                             
  E11000 duplicate key error collection: ****.orders index: _id_ dup key: { : "****-********-********-************" } ---> MongoDB.Driver.MongoBulkWriteException`1[MongoDB.Bson.BsonDocument]: A bulk write operation resulted in one or more errors.                                                                                                              
  E11000 duplicate key error collection: ****.orders index: _id_ dup key: { : "****-********-********-************" }                                                                                                                  
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)                                                                                                               
   at MongoDB.Driver.MongoCollectionBase`1.ReplaceOne(FilterDefinition`1 filter, TDocument replacement, UpdateOptions options, CancellationToken cancellationToken)                                                                                       
   --- End of inner exception stack trace ---                                                                                
   at MongoDB.Driver.MongoCollectionBase`1.ReplaceOne(FilterDefinition`1 filter, TDocument replacement, UpdateOptions options, CancellationToken cancellationToken)                                                                                       
   at Dashboard.Backend.AccountMonitor.ProcessOrder(OrderField& order)                                                       
   at Dashboard.Backend.AccountMonitor.OnRtnOrder(Object sender, OrderField& order)                                          
   at XAPI.Callback.XApi._OnRtnOrder(IntPtr ptr1, Int32 size1)                                                               
   at XAPI.Callback.XApi.OnRespone(Byte type, IntPtr pApi1, IntPtr pApi2, Double double1, Double double2, IntPtr ptr1, Int32 size1, IntPtr ptr2, Int32 size2, IntPtr ptr3, Int32 size3)                                                                   
Aborted (core dumped)

My question is, why is it possible to have dup key when I use ReplaceOne with Upsert = true options?

The app is working in the following environment and runtime:

.NET Command Line Tools (1.0.0-preview2-003121)

Product Information:
 Version:            1.0.0-preview2-003121
 Commit SHA-1 hash:  1e9d529bc5

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  16.04
 OS Platform: Linux
 RID:         ubuntu.16.04-x64

And MongoDB.Driver 2.3.0-rc1.

notionquest · Answer 1 · 2016-09-13T08:23:32.437

13

Upsert works based on the filter query. If the filter query doesn't match, it will try to insert the document.

If the filter query finds the document, it will replace the document.

In your case, it could have gone in either way i.e. insert/update. Please check the data to analyze the scenario.

Insert scenario:-

The actual _id is created automatically by upsert if _id is not present in filter criteria. So, _id shouldn't create uniqueness issue. If some other fields are part of unique index, it would create uniqueness issue.

Replace scenario:-

The field that you are trying to update should have unique index defined on it. Please check the indexes on the collection and its attributes.

Optional. When true, replaceOne() either: Inserts the document from the replacement parameter if no document matches the filter. Replaces the document that matches the filter with the replacement document.

To avoid multiple upserts, ensure that the query fields are uniquely indexed.

Defaults to false.

MongoDB will add the _id field to the replacement document if it is not specified in either the filter or replacement documents. If _id is present in both, the values must be equal.

edited Sep 13 '16 at 08:23

answered Sep 13 '16 at 08:13

notionquest

37,595
6
111
105

2

My filter is only on `_id` and the document to upsert/replace has exactly the same `_id`. – Kun Ren Sep 13 '16 at 08:16
1

Did you check the fields that you are trying to update whether any unique index defined on it? – notionquest Sep 13 '16 at 08:18
The pattern is that the app receives order records from the server. Each order has a unique id and will return several times with unchanged or changed status. Each time the app receives an order, it calls `ReplaceOne` to insert or replace to ensure that the database stores the latest data of all orders received. To my knowledge, there shouldn't be dup-key error in this case at all. – Kun Ren Sep 13 '16 at 08:22
Then, you need to check the source stream. I think it is sending some duplicate data. The replaceOne adds the new _id only if _id is not present in filter or replacement document. I have updated the same in the answer. I don't see any other issue in replaceOne. Worth to check the source feed to ensure that it is not sending dup data. – notionquest Sep 13 '16 at 08:25
You mean if the server sends data that is exactly the same with the existing document in the database, it will throw dup-key error? For example, in db, the doc is `{ _id: 1, a: 2, b: 3}`, `replaceOne` with exactly the same document would produce an error? – Kun Ren Sep 13 '16 at 08:27
I try with a small demo that `ReplaceOne` with exactly the same data and it does not throw any error. Only the result has `ModifiedCount = 0` – Kun Ren Sep 13 '16 at 08:39
in my case, even when i dont specify the ID during replace, it makes id `000000000000000000000000`, and on another upsert, it still makes the same id `000000000000000000000000` and it throws. – Seiko Santana Dec 03 '22 at 08:04

score 2 · Answer 2 · answered May 01 '20 at 15:22

I could not get IsUpsert = true to work correctly due to a unique index on the same field used for the filter, leading to this error: E11000 duplicate key error collection A retry, as suggested in this Jira ticket, is not a great workaround.

What did seem to work was a Try/Catch block with InsertOne and then ReplaceOne without any options.

try
{
    // insert into MongoDB
    BsonDocument document = BsonDocument.Parse(obj.ToString());
    collection.InsertOne(document);
}
catch
{
    BsonDocument document = BsonDocument.Parse(obj.ToString());
    var filter = Builders<BsonDocument>.Filter.Eq("data.order_no", obj.data.order_no);
    collection.ReplaceOne(filter, document);
}

score 0 · Answer 3 · answered Sep 13 '16 at 08:04

There is not enough information from you, but probably the scenario is the following: You receive data from server, replaceOne command doesn't match any record and try to insert new one, but probably you have a key in a document that is unique and already exists in a collection. Review and make some changes in your data before trying to update or insert it.

score 0 · Answer 4 · answered Jul 07 '17 at 07:55

I can co-sign on this one:

    public async Task ReplaceOneAsync(T item)
    {
        try
        {
            await _mongoCollection.ReplaceOneAsync(x => x.Id.Equals(item.Id), item, new UpdateOptions { IsUpsert = true });
        }
        catch (MongoWriteException)
        {
            var count = await _mongoCollection.CountAsync(x => x.Id.Equals(item.Id)); // lands here - and count == 1 !!!
        }
    }

score 0 · Answer 5 · answered Jul 28 '20 at 12:06

0

There is a bug in older MongoDB drivers, for example v2.8.1 has this problem. Update your MongoDB driver and the problem will go away. Please note when you use a new driver the DB version also needs to be updated and be compatible.

answered Jul 28 '20 at 12:06

jman

355
3
8

ReplaceOne throws duplicate key exception

5 Answers5