E11000 duplicate key error collection index: _id_ in mongo driver

Question

I am using InsertMany method in my service to insert 1000 documents in to MongoDB(in AWS DocumentDB). It is working fine when I use one pod in AWS. But If I scale my service to 2 or more pods it throws E11000 duplicate key error collection: myDB index: _id_. Can anybody help to resolve this issue?

Answer 1

This means you are trying to insert a document that clashes with an existing unique index, in this case it's caused by the _id field.

Without seeing the code it's hard to say exactly what's happening but as you mentioned it happens when you scale up it seems that those 2 services are each trying to insert all the documents, and once you try to insert a document that's already been inserted this error is thrown.

You should fix your scaling methods so that each process runs on it's own share of the documents so you're not trying to insert the same document twice.