How to hdf5 (Hdfsl ) file (One Column Read) read (Big Size file)

Question

I am using the HDF5DotNet with C# and I can read only full data as the attached image in the dataset. The hdf5 file is too big, up to nearly 1.4GB, and if I load the whole array into the memory then it will be out of memory.

I would like to read all data from One columns

double[] values = new double[203572];

string m_Doc_01 = "data/sample/line";

HDFql.Execute("USE DIRECTORY " + "\"" + File_Directory + "\""); 
HDFql.Execute("USE FILE " + "\"" + File_Name + "\"");
HDFql.Execute("CREATE CHUNKED(1, 203572) DATASET my_dataset_BS AS DOUBLE(2050, 203572)");

How to "m_Doc_01 ==> my_dataset_BS" Data ??? ???

for (int i = 0; i < 2050; i++)
    {
      HDFql.Execute("SELECT FROM " + "\"" + m_Doc_01 + "\"" + "(1:::1)  INTO MEMORY " + HDFql.VariableRegister(values));
    }

enter image description here

Please edit your question so it will be readable. – dpapadopoulos Aug 26 '19 at 05:13 — dpapadopoulos, Aug 26 '19 at 05:13

SOG · Answer 1 · 2019-08-28T05:29:32.573

0

To read the column that you have highlighted in the screenshot (i.e. column #0), you have to change the hyperslab to (please note the 0):

HDFql.Execute("SELECT FROM " + "\"" + m_Doc_01 + "\"" + "(, 0:::1)  INTO MEMORY " + HDFql.VariableRegister(values));

That said, if you want to loop through the dataset and read one column at the time do the following (also better to register variable values before the loop starts and unregister it after the loop is done - this will increase performance):

number = HDFql.VariableRegister(values);

for(int i = 0; i < 2050; i++)
{ 
    HDFql.Execute("SELECT FROM " + "\"" + m_Doc_01 + "\"" + "(, " + i + ":::1)  INTO MEMORY " + number);

    // do something with variable "values" (which contains the values of column #i)
}

HDFql.VariableUnregister(values);

edited Aug 28 '19 at 05:29

answered Aug 26 '19 at 12:01

SOG

876
6
10

SELECT FROM "data/sample/line"(0:::1) INTO MEMORY 0 Error message ( -99 ) HDFQL_ERROR_UNKNOWN Represents an operation that failed due to an unknown/unexpected error However, Low capacity files do not have errors. < 500 MB or less > – Tony Aug 27 '19 at 02:38
Could you turn on the debug mechanism (by executing operation `SET DEBUG ENABLE` in HDFql) and post here the output when you execute the operation that gives you an HDFQL_ERROR_UNKNOWN? – Aug 27 '19 at 12:49
script message – Tony Aug 28 '19 at 01:20
"Among the sample files I have " Error Count Row < 74470 , 79800 , 203572 , 259748 > // No Error Count Row < 564 , 2701 , 2714 , 10366 , 10367 > – Tony Aug 28 '19 at 01:57

How to hdf5 (Hdfsl ) file (One Column Read) read (Big Size file)

1 Answers1