Exchanging big lists between C# and F#

Question

I have big data files of historical stock data, that I want to load over a C# GUI and then process with different algorithms in F#. Additional values are appended to lists over time.

The lists are two-dimensional and in the form

[[id, Open,High,Low,Close], [id, Open,High,Low,Close], ...]

The F# code is in a library and cannot work with C# lists (I cannot get the values of the second dimension of the list in F#). Converting the entire list every time a F# operation takes place, is too expensive. The values cannot be stored in a F# list either, because C# can't handle them.

Proposed solutions are:

either storing an F# list in C#, just for storage purposes since the library can't store values, together with a C# list containing the same values, converting parts of the list as needed. The main problem here is that the files can be multiple GB in size.
reading the values that are needed directly from a file in F# and saving calculated values in additional files. The problem here is the efficient organisation of the files, so that a selection of lines (e.g. for a moving average calculation) to load can be done quickly.

Any more details on how you can't access the nested values? Is it a `System.Generics.List`? — wmeyer, Dec 19 '12 at 13:44
F# can work C# lists just fine (e.g. F# `seq` is just an alias for `IEnumerable`). And C# can work with F# lists too, you just need to reference `FSharp.Core` and then you can for example use `Microsoft.FSharp.Collections.ListModule.OfSeq()`. — svick, Dec 19 '12 at 14:02
The type in C# is `System.Generics.List>` and is passed as `IEnumerable` to F#. Isn't calling `OfSeq()` converting the list value, forcing me to loop through the entire list calling `OfSeq()` on every element and thus duplicating it. Since I have a two dimensional list, doing it this way means I have to first convert the list and then each element on its own. — user1915872, Dec 19 '12 at 14:24
Can you use `Seq.cast`? E.g. `Seq.cast> enumerable` (untested) — wmeyer, Dec 19 '12 at 14:33
Is there some reason you need a list instead of an array? C# arrays also have full F# support. — N_A, Dec 19 '12 at 15:21

score 3 · Accepted Answer · answered Dec 19 '12 at 15:17

Does the inner sequence need to be a list? If it always has the same elements (OHLC) it would be better to use a tuple (or a custom type) since it makes explicit its fixed nature. I would probably do something like this:

module Stock =
  let (!) s = DateTime.Parse(s)

  let getOHLC() : unit -> seq<DateTime * float * float * float * float> =
    seq [
      !"18-Dec-12",27.25,27.63,27.14,27.56
      !"17-Dec-12",26.79,27.22,26.68,27.10
      !"14-Dec-12",27.11,27.13,26.70,26.81
      //...
    ]

If you really need each element to be a list, you can pipe it through Seq.map seq.

The seq function is defined as:

let seq (x:seq<'T>) = x

so it only upcasts, not copy.

I had a small mistake in it and am now using tuples. Thanks so much! — user1915872, Dec 28 '12 at 15:59

score 1 · Answer 2 · answered Dec 22 '12 at 23:31

Converting the entire list every time a F# operation takes place, is too expensive.

Have you measured it?

The values cannot be stored in a F# list either, because C# can't handle them.

You should be able to enumerate over the elements of an F# list from C# because it is just another enumerable.

The type in C# is System.Generics.List<System.Generics.List<Double>> and is passed as IEnumerable to F#. Isn't calling OfSeq() converting the list value, forcing me to loop through the entire list calling OfSeq() on every element and thus duplicating it.

Calling something like Array.ofSeq would create a new collection but you don't need to do that to iterate over the contents.

Note that the F# name for the type you gave is ResizeArray<ResizeArray<float>>.

For example, you can do:

let f (xss: ResizeArray<ResizeArray<float>>) =
  for xs in xss do
    for x in xs do
      printfn "%A" x

to consume your data from F# without copying.

Exchanging big lists between C# and F#

2 Answers2