0

I have big LiteDB collection of items. I just want to find last one added by some criteria. Criteria are in lambda expression. My problem is, that it took realy long time (now around 2-3sec) and I just have testing database. In full mode, there will be 10 times more values.

My code now, with time stamps: 

var values = db.GetCollection<ExtendedValue>(IncomingValueCollection);
if (values.Count() > 0) {

    Stopwatch sw = new Stopwatch();
    sw.Reset();
    sw.Start();
    var expressedVals = values.Find(lambda);
    sw.Stop();
    Debug.WriteLine("find values " +  sw.ElapsedMilliseconds);

    sw.Reset();
    sw.Start();
    var inOrder = expressedVals.OrderByDescending(x => x.TimeCaptured).Take(5);
    sw.Stop();
    Debug.WriteLine("order " + sw.ElapsedMilliseconds);

    sw.Reset();
    sw.Start();
    var result = inOrder.First();
    sw.Stop();
    Debug.WriteLine("last " + sw.ElapsedMilliseconds);

    return result; 
    //withouth stopwatch
    //return values.Find(lambda).OrderByDescending(x => x.TimeCaptured).First();
}

and result is: 

find values 0
order 0
last 2399

So it is clear, that pick value take most of time.

I have tried:

  • Just use .First() on orderedEnum, no change
  • use Take(x) or not, no change
  • Convert it to Array or List and then pick first one, this take little longer, but nothing significant

I think, that .First() create array from enum and then pick first item. Is there any way to pick item withouth generating whole array and creating big lag?

EDIT: According to comments from Matthew Watson, i edited code, to find out, that my lambda expression take most of time and conclusion, that First() methot takes most time is mistake. It takes only becaouse it execute whole query at once.

if (values.Count() > 0) {
    Stopwatch sw = new Stopwatch();
    sw.Reset();
    sw.Start();
    var expressedVals = values.Find(lambda).ToList();
    sw.Stop();
    Debug.WriteLine("find values " +  sw.ElapsedMilliseconds);
    sw.Reset();
    sw.Start();
    var inOrder = expressedVals.OrderByDescending(x => x.TimeCaptured).ToList();
    sw.Stop();
    Debug.WriteLine("order " + sw.ElapsedMilliseconds);
    sw.Reset();
    sw.Start();
    var result = expressedVals.First();
    sw.Stop();
    Debug.WriteLine("last " + sw.ElapsedMilliseconds);

    return result; 
    //withouth stopwatch
    //return values.Find(lambda).OrderByDescending(x => x.TimeCaptured).First();

with result

find values 2395
order 4
last 0

So I think, that I cannot do it this way, I have to create workaround.

Uwe Keim
  • 39,551
  • 56
  • 175
  • 291
110mat110
  • 564
  • 6
  • 26
  • When only the first item is required then why are you taking 5 records first? Reason why the last statement taking time because, on that statement only the whole formed query is executed against the persistent storage, before that everything is `IQueryable` result which is not executed until requested to execute which is the `First` operation. – user1672994 Oct 06 '19 at 08:36
  • You are actually timing how long it takes to sort the data, since you are initialising `inOrder` to an `IEnumerable` that hasn't yet been executed. Only when you call `inOrder.First()` is it actually executed. But what is the exact type of `expressedVals`? Is that also an Enumerable? – Matthew Watson Oct 06 '19 at 08:37
  • So if you only want to find the maximum single value (rather than the maximum N values) then you don't need to sort the data at all - you just need to make a single pass to find the maximum value. Have a look at [MoreLinq/s `MaxBy()`](https://github.com/morelinq/MoreLINQ/blob/master/MoreLinq/MaxBy.cs) (available via NuGet) for this, or just write a loop. – Matthew Watson Oct 06 '19 at 08:40
  • "Only when you call inOrder.First() is it actually executed" That means, that I do not know what execution take long time. I could be ordering, that could be lambda or whatever else. I will try MoreLinq and also there is LiteDB orders, that are little more complicated than lamba. And no matter how many I take, it could be 1, 5, 100 it always take same time – 110mat110 Oct 06 '19 at 08:46
  • 1
    `no matter how many I take, it could be 1, 5, 100 it always take same time` Yes, because you are always sorting the entire set of data before taking them. – Matthew Watson Oct 06 '19 at 08:48
  • Since LiteDB supports indexes if you use its native Query capabilities, is it not possible to index TimeCaptured and then Find on that index? – NetMage Oct 07 '19 at 19:47

0 Answers0