Copying data from a LiteDb Query into a List causes System.InvalidOperationException

Question

I've been playing around with LiteDb to improve the performance of my Data Feed. The feed runs on an Azure database and in order to reduce the amount DTU's and connections we use(I'm doing some logic inside a parallel forach loop), I pull the data from the database, put it into LiteDb index it and then run my logic around it. Most of my scenarios work great. I first do an insert bulk add indexes and then reopen litdb in readonly mode and run my queries Something like this

using (var db = new LiteDatabase("SiloLite.db"))
                {
                    db.DropCollection("SiloProduct");
                    var products = db.GetCollection<SiloProduct>("SiloProduct");
                    products.InsertBulk(_siloproductRepository.Table.ToList());
                    products.EnsureIndex(x => x.Sku);
                    db.Engine.EnsureIndex("SiloProduct", "UniqueProdId", "$.Sku+';'+$.ParentProductId", true);
                }
using (var db = new LiteDatabase($"Filename={AppDomain.CurrentDomain.BaseDirectory}\\SiloLite.db;mode=ReadOnly"))
                {
                    var products = db.GetCollection<SiloProduct>("SiloProduct");
                    var manufacturers = db.GetCollection<SiloManufacturer>("SiloManufacturer");
                    var productspecificationmap =
                        db.GetCollection<SiloProductSpecificationAttributeMapping>(
                            "SiloProductSpecificationAttributeMapping");
                    var specificationAttributes =
                        db.GetCollection<SiloSpecificationAttribute>("SiloSpecificationAttribute");
                    var specificationAttributeOptions =
                        db.GetCollection<SiloSpecificationAttributeOption>("SiloSpecificationAttributeOption");

                    Parallel.ForEach(katartlist, KatartItem =>
                    {
                        SaveData(KatartItem, specificationAttributes, specificationAttributeOptions, productspecificationmap,
                            manufacturers, products);
                    });
                    _logger.Information("Completed updating Product Specifications");
                }

And it works great in all scenarios exceot one. If I try to run a query and then try to save that data into another collection LiteDb complains. This is what I'm trying to do

var currentspecOptCollection = new List<SiloProductSpecificationAttributeMapping>(liteProdSpecCollection.Find(x => x.ProductId.Equals(productId)).ToList());
                    var specstoDelete = currentspecOptCollection
                                                          .Where(x => !specificationattributeoptionlist.Contains(x.SpecificationAttributeOptionId))
                                                          .ToList();
                    if (specstoDelete.Any())
                    {
                        foreach (var specattr in specstoDelete)
                        {
                            _specattrDeleteList.Add(specattr);
                        }

                    }

This logic is executed inside the Parallel.Foreach and LiteDb throws this error

System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
   at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)
   at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value)
   at LiteDB.CacheService.GetPage(UInt32 pageID)
   at LiteDB.PageService.GetPage[T](UInt32 pageID)
   at LiteDB.QueryCursor.Fetch(TransactionService trans, DataService data, BsonReader bsonReader)
   at LiteDB.LiteEngine.Find(String collection, Query query, Int32 skip, Int32 limit)+MoveNext()
   at LiteDB.LiteEngine.Find(String collection, Query query, String[] includes, Int32 skip, Int32 limit)+MoveNext()
   at LiteDB.LiteCollection`1.Find(Query query, Int32 skip, Int32 limit)+MoveNext()
   at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)
   at CreateOrUpdateProductSpecifications(KatartItem katartItem, VarData vd, SiloManufacturer manufacturer, SiloProduct product, LiteCollection`1 liteSpecCollection, LiteCollection`1 liteSpecOptCollection, LiteCollection`1 liteProdSpecCollection) in *** 1399

Not sure what I'm doing wrong here.Any pointers will help

Muhammad Ali Nazar · Answer 1 · 2019-08-05T21:28:45.303

I had to solve a similar problem in which i was reading data from LiteDB and then putting the data into another list. I solved it by writing my parallel execution like this:

public static async Task LoadDataFromDB()
{
    var tasks = new List<Task>();

    using (var db = new LiteDatabase(fullDbPath))
    {
        var bookChapterCollection = db.GetCollection<Chapter>("bookData");
        foreach (var volume in Volume.ListOfVolumes)
        {
            tasks.Add(Task.Factory.StartNew(() =>
            {
                var volumeChapterFromDB = bookChapterCollection.Find(x => x.chapterVolume == volume.volume).toList();
                return volumeChapterFromDB;
            }
            ,TaskCreationOptions.LongRunning)
           .ContinueWith(task => {
                BookChaptersBag.Add(task.Result);
                return task.Result;
            })
            );
        }
        await Task.WhenAll(tasks).ContinueWith(task =>
        {
            BookChaptersBag.CompleteAdding();
        });
    }
}

Here the BookChaptersBag is a BlockingCollection<List<Chapter>> which is a thread safe collection from System.Collections.Concurrent namespace. After writing my parallel execution like above i stopped getting the exception.

Hope this helps!

Copying data from a LiteDb Query into a List causes System.InvalidOperationException

1 Answers1