C#. Fastest way to save list of Objects to Mongo

Question

I have several lists of Objects

List<ObjectClass1> ObjectList1;
List<ObjectClass2> ObjectList2;

I would like to write all objects as JSON to Mongo in the end of test run. What is the fastest way to do this? I am currently doing this:

IMongoClient client = new MongoClient();
IMongoDatabase db = client.GetDatabase("MyDB");
db.CreateCollection("ObjectList1");
var ObjectList1Collection = db.GetCollection<BsonDocument>("ObjectList1");
foreach(ObjectClass1 obj in ObjectList1)
{
   var document = BsonSerializer.Deserialize<BsonDocument>(MyJSONSerializer.Serialize(obj));
   ObjectList1Collection.InsertOneAsync(document);          
}

db.CreateCollection("ObjectList2");
var ObjectList1Collection = db.GetCollection<BsonDocument>("ObjectList2");
foreach(ObjectClass2 obj in ObjectList2)
{
   var document = BsonSerializer.Deserialize<BsonDocument>(MyJSONSerializer.Serialize(obj));
   ObjectList2Collection.InsertOneAsync(document);          
}

score 5 · Accepted Answer · answered Jul 31 '17 at 21:22

May I suggest you start with the following code:

IMongoClient client = new MongoClient();
IMongoDatabase db = client.GetDatabase("MyDB");
// create collection calls are not needed, MongoDB will do that for you
// db.CreateCollection("ObjectList1");
var objectList1Collection = db.GetCollection<ObjectClass1>("ObjectList1");
objectList1Collection.InsertMany(ObjectList1);

...and more or less the same for the second list of objects. This will simply run the insert in a bulk load fashion, i.e. avoid the overhead of calling MongoDB thousands of times and instead chunk up your list of objects into packages of 1000 documents and send them to MongoDB.

If that's not fast enough, there's various things that might make sense depending on your setup:

Profile what's going on! There's little point in optimizing as long as you don't know what the bottleneck is.
The serialization process (conversion of your entities to BsonDocuments) is pretty hefty in terms of CPU power needed so you would want to do that bit in parallel (using multiple threads) - you would want a CPU with a lot of cores for that.
Then you'd want to use the async implementation of the InsertMany method mentioned above so your CPU can continue working while its waiting for the network/IO part after sending a chunk of documents off to MongoDB.
You should try to keep your documents as tiny as possible if you're after raw performance - never underestimate the impact of that aspect!
You can invest into stronger hardware. That's always an option...
You can do various things around the MongoDB setup, including sharding to distribute the load to various systems if the I/O part is the culprit.
You can play around with write concern levels
You can fiddle with the MongoDB storage engine
...and probably a lot more dangerous stuff. ;)

score 2 · Answer 2 · answered Jul 31 '17 at 18:58

2

You don't need to serialize it to Json you can just call:

ObjectList1Collection.InsertManyAsync(ObjectList1);

That should be te fastest way as far as I know.

answered Jul 31 '17 at 18:58

BOR4

610
4
9

C#. Fastest way to save list of Objects to Mongo

2 Answers2