MongoDB BulkWrite ExceededTimeLimit error in .Net

Question

I'm trying to push about 150k updates into Mongo database (v 4.2.9 running on Windows, stage replica with two nodes) using BulkWrite on c# driver (v2.11.6) and looks like it is impossible. The project is .Net Framework 4.7.2.

Mongo c# driver documentation is terrible, but somehow on forums and with a lot of googling, I was finnaly able to find a way how to run about 150k updates using a batch, something like this (a little simplified for SO):

client = new MongoClient(connString);
database = client.GetDatabase(db);

// Build all the updates
List<UpdateOneModel<GroupEntry>> updates = new List<UpdateOneModel<GroupEntry>>();
foreach (GroupEntry groupEntry in stats)
{
  FilterDefinition<GroupEntry> filter = Builders<GroupEntry>.Filter.Eq(e => e.Key, groupEntry.Key);

  UpdateDefinitionBuilder<GroupEntry> update = Builders<GroupEntry>.Update;
  var groupEntrySubUpdates = new List<UpdateDefinition<GroupEntry>>();

  if (groupEntry.Value.Clicks != 0)
    groupEntrySubUpdates.Add(update.Inc(u => u.Value.Clicks, groupEntry.Value.Clicks));

  if (groupEntry.Value.Position != 0)
    groupEntrySubUpdates.Add(update.Set(u => u.Value.Position, groupEntry.Value.Position));

  UpdateOneModel<GroupEntry> groupEntryUpdate = new UpdateOneModel<GroupEntry>(filter, update.Combine(updates));
  groupEntryUpdate.IsUpsert = true;

  updates.Add(groupEntryUpdate);
}

// Now BulkWrite them in transaction to make sure data are consistent
IClientSessionHandle session = client.StartSession();
session.StartTransaction();

IMongoCollection<GroupEntry> collection = database.GetCollection<GroupEntry>(collectionName);

// Following line FAILS after some time
BulkWriteResult<GroupEntry> bulkWriteResult = collection.BulkWrite(session, updates);

if (!bulkWriteResult.IsAcknowledged)
  throw new Exception("Mongo BulkWrite is not acknowledged!");

session.CommitTransaction();

The problem is that I keep getting the following exception:

{
   "operationTime":Timestamp(1612737199,
   1),
   "ok":0.0,
   "errmsg":"Exec error resulting in state FAILURE :: caused by :: operation was interrupted",
   "code":262,
   "codeName":"ExceededTimeLimit",
   "$clusterTime":{
      "clusterTime":Timestamp(1612737199,
      1),
      "signature":{
         "hash":new BinData(0,
         "ljcwS5Gf2JBpEu/OgPFbvRqclLw="")",
         "keyId":"NumberLong(""6890288652832735234"")"
      }
   }
}

Does anyone have any clue? Mongo c# driver docs are completely useless. It looks like I should somehow set property $maxTimeMS, but it is not possible on BulkInsert. I have tried:

Restarts and rebuilds
Different versions of MongoDriver
Set much bigger timeouts for all "timeout" properties on MongoClient and session
Create smaller batches for BulkWrite (up to 1000 items per batch). Fails after 50-100 updates.
Spent hours and hours in useless Mongo docs and Mongo JIRA

So far no luck. The funny thing is, that the same approach works on c# driver 2.10.3 on .Net CORE 3.1 (yes, i tried) even with bigger batches (about 300k updates).

What am I missing?

EDIT:

I tried set maxCommitTime to 25 minutes based on dododo's comments like this:

IClientSessionHandle session = client.StartSession(new ClientSessionOptions()
{
  DefaultTransactionOptions = new TransactionOptions(new Optional<ReadConcern>(ReadConcern.Default), 
    new Optional<ReadPreference>(ReadPreference.Primary), 
    new Optional<WriteConcern>(WriteConcern.Acknowledged), 
    new Optional<TimeSpan?>(TimeSpan.FromMinutes(25)))
});

It now throws exception while doing commmit: NoSuchTransaction - Transaction 1 has been aborted.. We checked MongoDB log file and found new error in there:

Aborting transaction with txnNumber 1 on session 09ea7755-7148-43e8-83d8-8bf58c211bda because it has been running for longer than 'transactionLifetimeLimitSeconds

Based on docs, this is 60 seconds by default. So we set it to 5 minutes and now it works.

So, thank you dododo for pointing me the right direction.

Anyway, it would be really great if Mongo team described errors better and write documentation above basic CRUD operations.

it's a server error (not .NET driver) and it looks like `maxTimeMS` works only with cursors. I suspect it's a transaction behavior and not bulk operations itself, so presumably, your operation works without session. If so, try to set `maxCommitTime` in transaction options — dododo, Feb 08 '21 at 00:00
`that the same approach works on c# driver 2.10.3 on .Net CORE 3.1` - this looks really suspicious, can you make more attempts? Like check this case once again for the same database version, the same data structure/records with the same indexes for both net framework and net core for 2.10.3 and 2.11.6? — dododo, Feb 08 '21 at 00:09
Even if you bash the documentation you still should be including the relevant information you have located in it. — D. SM, Feb 08 '21 at 02:42
dododo I tried your recomendation and updated the question accordingly. Thanks. Regarding .Net Core 3.1 - I'm not able to do what you suggest (stage/prod env), but based on the resolution it looks like production is quick enought to finish each transaction in 60 seconds. That is why it works there. D. DM : I added link to a page where this approach is described. — Martin Brabec, Feb 08 '21 at 14:32

score 0 · Answer 1 · answered Feb 08 '21 at 14:42

As dododo suggested, this error was manifestation of server closing the transaction, because it took longer then transactionLifetimeLimitSeconds, which is 60 seconds by default. So two things needs to be done:

Set parameter transactionLifetimeLimitSeconds to more than 60 seconds
Set maxCommitTime to higher value. I'm unable to find default value, so I set it to 10 minutes (same as transactionLifetimeLimitSeconds). Set it while starting a session (see the question).

Anyway documentation for this is missing and the error itself was misleading. So I hope it helps anyone who will have to deal with with this.

MongoDB BulkWrite ExceededTimeLimit error in .Net

1 Answers1