I want to read a large bin file which contains M successions of double numbers, which repeat in a predefined order as:
A0, A1, A2... AN, B0, B1, B2... BN... (M times)
I am not allowed to read the file at once because the file can be too large, that it cannot fit the main memory. Therefore I use the classes RandomAccessFile
and MappedByteBuffer
in order to divide the file in slices (channels). Given that I know the number of numbers in each succession (N) but I do not know the total number of sucessiions (M), I would like to know what is the best practice regarding the following cases:
1) What is the best way to save the data? I use a List<List<Double>>
since I think Map does not provide flexibility (it would be too slow) to tackle this case? Am I right?
2)What is the way to determine, that the end of the file is reached and the algorithm should stop?
Here is the Code I have written:
public void parseFile() throws IOException{
RandomAccessFile raf = new RandomAccessFile(file, "r");
long pos = raf.getFilePointer();
while((pos = raf.getFilePointer()) != -1){
parseDataSets(pos, 20);
}
raf.close();
}
private void parseDataSets(long pos, long size)
throws IOException {
List<List<Double>> valuesSet = new ArrayList<List<Double>>();
mbb = raf.getChannel().map(FileChannel.MapMode.READ_ONLY, pos,
size * 8 * numberOfSignals);
for (int i = 0; i < size; i++) {
for (int j = 0; j < numberOfSignals; j++) {
if (valuesSet.get(j) == null)
valuesSet.set(j, new ArrayList<Double>());
if (j == 0)
valuesSet.get(j).add((double) mbb.getDouble());
else
valuesSet.get(j).add(mbb.getDouble());
}
}
}
I think it needs a while loop in the method parseDataSets
to check if the end of the file is reached earlier (if the last slice contains less than 20 successions).
Update: The first number of each succession is long, not double, therefore the if-else check in the parseDataSets
method.