How to load big CSV file in ILArray for 3d plotting?

Question

I have several files in csv format with scientific data to examine plotting them in 3D. The file content is organized as datetime, frequency in Hz and intensity in dB as in this example:

2014-03-15 18:00:00, 19700000.0, -30.19
2014-03-15 18:00:00, 19700781.25, -31.19
2014-03-15 18:00:00, 19701562.5, -30.43
2014-03-15 18:00:00, 19702343.75, -30.37
2014-03-15 18:00:00, 19703125.0, -27.06

For the plotting context, the 3 values represent respectively x,y,z.

I would like to create a custom type named datasample consisting of datetime,float,float types and then declare the array as

ILArray<datasample>

Is there a specific function or suggested way of loading from file the ILArray for plotting ?

Can ILNumerics correctly handle the datetime format of the first column or have I to preprocess the file to convert it to something else ?

Thank you for your kind support

UPDATED after the answer with the following source code I've loaded 3.6 Million points with this and the 3d plot is rendered smoothly. I would need some instructions on how to color the individual points based on their Z value. Seems that using a surface with millions of point in input is a bit...heavy :-) Any performance hints I could use ?

This is a screenshot:

using ILNumerics;
using ILNumerics.Drawing;
using ILNumerics.Drawing.Plotting;

using System.IO;
using System.Globalization;

private void ilPanel1_Load(object sender, EventArgs e)
{

    fsin = new System.IO.StreamReader("testlong.iln");

    while(fsin.EndOfStream == false)
    {
        datarow = fsin.ReadLine();
        // Console.WriteLine(datarow);

        rowvalues = datarow.Split(',');
        inrows = inrows + 1;

        tempX.Add(float.Parse(rowvalues[0], CultureInfo.InvariantCulture));
        tempY.Add(float.Parse(rowvalues[1], CultureInfo.InvariantCulture));
        tempZ.Add(float.Parse(rowvalues[2], CultureInfo.InvariantCulture));

    }
    fsin.Close();

    // now that I know the input size (number of x,y,z tuples), I can 
    // create and initialize the ILArray :
    ILArray<float> datasamples = ILMath.zeros<float>(3, (int)inrows );

    label1.Text = String.Format("Data points read: {0:N0}", inrows);

    // ... and now I can copy data from the input arrays:
    datasamples["0;:"] = tempX.ToArray();
    datasamples["1;:"] = tempY.ToArray();
    datasamples["2;:"] = tempZ.ToArray();

    // these are no longer used so...
    tempX.Clear();
    tempY.Clear();
    tempZ.Clear();

    var scene = new ILScene {
        new ILPlotCube(twoDMode: false) {
            new ILPoints {
                Positions = datasamples,
                Color = Color.Blue,
                // Colors = 
                Size = 0.5F
            }
        }
    };

    // at the end of all modifications, call Configure()
    ilPanel1.Scene = scene;
    ilPanel1.Scene.Configure();
    ilPanel1.Refresh();
}

Answer 1

CSV data are read using ILMath.csvread<T>(). The function allows the specification of the overall element type via the generic parameter:

string s =
@"2014-03-15 18:00:00, 19700000.0, -30.19
2014-03-15 18:00:00, 19700781.25, -31.19
2014-03-15 18:00:00, 19701562.5, -30.43
2014-03-15 18:00:00, 19702343.75, -30.37
2014-03-15 18:00:00, 19703125.0, -27.06";

ILArray<double> A = ILMath.csvread<double>(s);

>A
<Double> [5,3]
(:,:) 1e+007 *
    0,00000    1,97000    0,00000
    0,00000    1,97008    0,00000
    0,00000    1,97016    0,00000
    0,00000    1,97023    0,00000
    0,00000    1,97031    0,00000

There is currently no individual column format specification. So, in order to include the dates as such (DateTime), you must create your own datatype – as you pointed out.

This, however, introduces other problems:

1) csvread() expects homogeneous elements from the csv file. While ILArray<datasample> is fine (and useful), csvread does not support that directly.

2) If your goal is to plot the data, you will have to convert the datetime to some real number anyway. The Drawing namespace expects ILArray only.

3) Furthermore, most ILMath functions are specialized for ILArray of some real (System.Value) element format. Having an ILArray would bring very limited support by means of those functions.

EDIT: How about this? :)

private void ilPanel1_Load(object sender, EventArgs e) { ILArray<float> A = ILMath.tosingle(ILMath.rand(4, 3000)); ilPanel1.Scene.Add( new ILPlotCube(twoDMode:false) { new ILPoints() { Positions = A["0,1,2;:"], Colors = new ILColormap(Colormaps.Jet).Map(A["2;:"]).T, Color = null } }); }