2

What is the recommended way to allocate large array in ILNumerics, reading large (>1 GB or >>1 GB) CSV file?

I am aware of the ILMath.csvread static class, is there a better way to handle multiple types on a single line? For example I may want to read a very long int (for dates) and doubles comma separated. Any efficient way of achieving this?

What about memory management issues? How to allocate the right memory size on the fly? Can I store everything in ILArrays? Should I define my own type to encapsulate several arrays?

To summarize: what is the recommended way of reading into memory a large number of csv rows (100 million rows or even billions)?

Thanks in advance

alea_iacta_est
  • 180
  • 9
  • `32bit` NET assemblies are restricted to `2Gb` max for each variable or array. You can get more allocated into memory when compiling for `64bit`. Just saying. – John Alexiou Jun 04 '14 at 23:16
  • 1
    I know for a fact that re-sizing an `ILArray<>` is really slow. Read your data in your own structures and use the `GetArrayForWrite()` methods to transfer the data to `ILArray` if needed. – John Alexiou Jun 04 '14 at 23:53

0 Answers0