0

I want to read a large bin file which contains M successions of double numbers, which repeat in a predefined order as:

A0, A1, A2... AN, B0, B1, B2... BN... (M times)

I am not allowed to read the file at once because the file can be too large, that it cannot fit the main memory. Therefore I use the classes RandomAccessFile and MappedByteBuffer in order to divide the file in slices (channels). Given that I know the number of numbers in each succession (N) but I do not know the total number of sucessiions (M), I would like to know what is the best practice regarding the following cases:

1) What is the best way to save the data? I use a List<List<Double>> since I think Map does not provide flexibility (it would be too slow) to tackle this case? Am I right?

2)What is the way to determine, that the end of the file is reached and the algorithm should stop?

Here is the Code I have written:

public void parseFile() throws IOException{

        RandomAccessFile raf = new RandomAccessFile(file, "r");
        long pos = raf.getFilePointer();
        while((pos = raf.getFilePointer()) != -1){
            parseDataSets(pos, 20);
        }

        raf.close();
    }

private void parseDataSets(long pos, long size)
            throws IOException {

        List<List<Double>> valuesSet = new ArrayList<List<Double>>();
        mbb = raf.getChannel().map(FileChannel.MapMode.READ_ONLY, pos,
                size * 8 * numberOfSignals);


            for (int i = 0; i < size; i++) {
                for (int j = 0; j < numberOfSignals; j++) {
                    if (valuesSet.get(j) == null)
                        valuesSet.set(j, new ArrayList<Double>());

                    if (j == 0)
                        valuesSet.get(j).add((double) mbb.getDouble());
                    else
                        valuesSet.get(j).add(mbb.getDouble());
                }
            }

    }

I think it needs a while loop in the method parseDataSets to check if the end of the file is reached earlier (if the last slice contains less than 20 successions).

Update: The first number of each succession is long, not double, therefore the if-else check in the parseDataSets method.

arjacsoh
  • 8,932
  • 28
  • 106
  • 166
  • 2
    If the file does not fit in memory, I don't think the resulting array of values will fit either (or, at least, it will be huge). – Stefano Sanfilippo May 07 '13 at 14:03
  • about 2, mbb.getLong() will throw BufferUnderflowException - If there are fewer than eight bytes remaining in this buffer – andy May 07 '13 at 14:14

0 Answers0