Is it correct that Deedle/Series is slow compared to a list?

Question

I am working on a data "intensive" app and I am not sure if I should use Series./DataFrame. It seems very interesting but it looks also way slower than the equivalent done with a List ... but I may not use the Series properly when I filter. Please let me know what you think.

Thanks

type TSPoint<'a> =
 {
    Date : System.DateTime
    Value : 'a            
 }

type TimeSerie<'a> = TSPoint<'a> list

let sd = System.DateTime(1950, 2, 1)
let tsd =[1..100000] |> List.map (fun x -> sd.AddDays(float x))

// creating a List of TSPoint
let tsList = tsd |> List.map (fun x -> {Date = x ; Value = 1.}) 
// creating the same as a serie
let tsSeries = Series(tsd , [1..100000] |> List.map (fun _ -> 1.))

// function to "randomise" the list of dates
let shuffleG xs = xs |> List.sortBy (fun _ -> Guid.NewGuid())

// new date list to search within out tsList and tsSeries
let d = tsd |> shuffleG |> List.take 1000

// Filter
d |> List.map (fun x -> (tsList |> List.filter (fun y -> y.Date = x)))
d |> List.map (fun x -> (tsSeries |> Series.filter (fun key _ -> key = x)))

Here is what I get:

List -> Real: 00:00:04.780, CPU: 00:00:04.508, GC gen0: 917, gen1: 2, gen2: 1

Series -> Real: 00:00:54.386, CPU: 00:00:49.311, GC gen0: 944, gen1: 7, gen2: 3

tbh, List itself will be very slow. Depending on your use case consider an array. — s952163, Nov 22 '18 at 08:07
but an array is mutable, which is something I am not a huge fan of — Jeff_hk, Nov 22 '18 at 08:11

score 3 · Accepted Answer · answered Nov 22 '18 at 11:43

In general, Deedle series and data frames do have some extra overhead over writing hand-crafted code using whatever is the most efficient data structure for a given problem. The overhead is small for some operations and larger for some operations, so it depends on what you want to do and how you use Deedle.

If you use Deedle in a way in which it was intended to be used, then you'll get a good performance, but if you run a large number of operations that are not particularly efficient, you may get a bad performance.

In your particular case, you are running Series.filter on 1000 series and creating a new series (which is what happens behind the scenes here) does have some overhead.

However, what your code really does is that you are using Series.filter to find a value with a specific key. Deedle provides a key-based lookup operation for this (and it's one of the things it has been optimized for).

If you rewrite the code as follows, you'll get much better performance with Deedle than with list:

d |> List.map (fun x -> tsSeries.[x])
// 0.001 seconds

d |> List.map (fun x -> (tsSeries |> Series.filter (fun key _ -> key = x)))
// 3.46 seconds

d |> List.map (fun x -> (tsList |> List.filter (fun y -> y.Date = x)))
// 40.5 seconds

thanks, I didn't think to proceed this way. I will use TryGet so I get returned a OptionalValue and so I can do some pattern matching instead of getting an exception (in case of vfailure to find the value). — Jeff_hk, Nov 23 '18 at 02:07
@John_hk Yes, `TryGet` is the way to go if the key or value might be missing! — Tomas Petricek, Nov 23 '18 at 22:07

Is it correct that Deedle/Series is slow compared to a list?

1 Answers1