I have read the following two questions but did not see a satisfactory solution or suggestion:
- Performance degrade with Mongo when using bulkwrite with upsert
- Mongodb/Mongoose bulkwrite(upsert) performance issues
I have an API that will be receiving ~80k objects every 5 - 6 seconds.
These objects will need to be inserted if they do not exist and updated if they do exist.
The collection has a unique index (of type asc 1), which I use in the filter expression below.
Upserting 100k documents using the code below takes more than a minute, which is too long for my requirement (instantiating the objects takes less than a second).
MongoClient _mongoClient = new MongoClient("connection string");
IMongoDatabase _hotsauceOdds = _mongoClient.GetDatabase("DatabaseName");
var collection = _hotsauceOdds.GetCollection<Person>("TestMongo");
List<Person> peeps = new List<Person>();
for (int i = 0; i < 100000; i++)
{
peeps.Add(new Person { Age = i, Name = Guid.NewGuid().ToString() });
}
var models = peeps.Select(item => new ReplaceOneModel<Person>(new ExpressionFilterDefinition<Person>(doc => doc.Name == item.Name), item) { IsUpsert = true });
await collection.BulkWriteAsync(models);
I am testing this on the free tier of mongo atlas.
Is it possible to make this operation run more quickly, or is this as good as it gets for mongodb?
