2

How does F#'s list handling in Deedle and List.fold compare to vectorized handling of pandas dataframes in python performance-wise? What are the mechanical differences under the hood?

I understand

  • Python's pandas's performance gain using vectorized functions stems from utilization of precompiled cython and C components keeping the heavy lifting close to the machine (with c-like performance).
  • F#'s Deedle provides precompiled functionality in the same fashion, but it is still written in F#. Does this compare performance-wise?

I would like to understand more of this, the F# handling of lists, and the different approaches within the two ecosystems.

harmonica141
  • 1,389
  • 2
  • 23
  • 27
  • 1
    `List.fold` is a native, straightforward implementation of fold, not suitable for high-performance or big-data applications. Try [Deedle](https://bluemountaincapital.github.io/Deedle/). – Fyodor Soikin Dec 27 '20 at 01:04
  • Thanks for pointing me to it. One can only find what he knows to look for. Though: https://github.com/fslaborg/Deedle tells me this is written in F# almost exclusively. Edited question because the core of it remains unanswered. – harmonica141 Dec 27 '20 at 13:09
  • To answer your question, you will need to measure the performance given the kind of data that you want to process. [This article](http://type-nat.ch/post/vectorize-all-the-things-multiply-add/) demonstrates using https://github.com/dotnet/BenchmarkDotNet to do that. It also demonstrates a way to do vectorization in .NET, which could be done using F#. The performance of any .NET solution might depend greatly on which framework you target, with the latest .NET 5 probably being the fastest. – Scott Hutchinson Dec 28 '20 at 18:32
  • 2
    Here is [another article](https://jackmott.github.io/programming/2016/07/22/making-obvious-fast.html) that compares several approaches in F# and C# with C. Note the `Vector` type and `Array.SIMD.fold`, which are just as fast as the C implementations. – Scott Hutchinson Dec 28 '20 at 18:40

0 Answers0