0

We have a tool in our C++ codebase that will output EEG float values into Tabbed delimited text. If we have, for example, 256 channels and 550000 samples, we output a CSV Tabbed test file that should open in Excel with 256 columns and 550000 rows.

When I try to open this file in Open Office (following the instructions for opening TDT files), I can see the progress bar working very slowly, eventually it finishes, but the file never opens.

When I try to open it in Excel 2011 for Mac, it hangs for several minutes and then I eventually get a dialog "Not Enough memory". I look at the activity monitor, and I have 16 gigs of memory, approx 2.7 being used, and by the time Excel is returning this message, I Still have about 6 gigs memory available. The size after this file is converted goes from about 560MB to 1.81 gigs, Which seems about right for the string size of the values.

Smaller files such as 100000 samples and 256 channels open fine and look correct.

According to Excel's docs, it should be able to display over 1 million rows and something like 16000 columns.

I wrote a Python script to check the file and it has 550,000 lines in it as it should. I also wrote some C++ test code to make sure it wasn't anything I was doing.

Code snippet:

// C style file stream.
#include <cstdio>
#include <iostream>

int main(int argc, const char * argv[])
{
    std::cout<<"Procesing"<<std::endl;
    std::string fileURI = "/Users/mmurphy/Desktop/TabbedTestFile.txt";
   float eeg = 87658.98765;

    // Create a C style file stream object.
    FILE* dataStream;

    dataStream = fopen(fileURI.c_str(), "w+");

    for(int32_t s = 0; s < 550000; s++){


        for(int32_t c = 0; c < 256; c++){

            fprintf(dataStream, "%f\t",eeg);
        }

        // Simulate the Vref Channel (257)
        fprintf(dataStream, "%f\n",0.0);

    }

    // Close the fileStream
    fclose(dataStream);

    std::cout<<"Complete"<<std::endl;

}

If I change the number of columns from 256 to say 3, it works fine. Should I assume since this is happening in both, Excel and OO, there may be some type of bug that was introduced with OS Mavericks?

Any information is appreciated.

ZygD
  • 22,092
  • 39
  • 79
  • 102
Miek
  • 1,127
  • 4
  • 20
  • 35
  • 1
    I would say that this is simply a too big a file, even if excel only allocates say 4 bytes per cell, thats already >500MB. Further, if excel is a 32 bit application it would stop allocation with out of memory error at something between 2 and 4 GB, regardless of your amount of memory installed. Given row/column counts in the docs are rather marketing... perhaps assuming that only a fraction of cells is used. Summing up, i don't think excel is the right tool here. Depending on your needs maybe try matlab 64bit. – Thomas Sep 21 '14 at 21:08
  • What do you want to do with the files ???. It sounds to be to big for a Spreadsheet. A database might be the answer, alternatively ReCsvEditor (http://recsveditor.sourceforge.net/) might be able to handle the file (you will probably need to modify the Shell script to give java more ram). – Bruce Martin Sep 22 '14 at 02:27
  • @Thomas Thanks. You are correct that the file will open in Matlab. It must be that MSExcel or OO ever actually created a 64 bit version of their spreadsheet software for Mac OS X – Miek Sep 22 '14 at 15:24

0 Answers0