How to read binary file in one block rather than using a loop in matlab

Question

I have this file which is a series of x, y, z coordinates of over 34 million particles and I am reading them in as follows:

parfor i = 1:Ntot
 x0(i,1)=fread(fid, 1, 'real*8')';
 y0(i,1)=fread(fid, 1, 'real*8')';
 z0(i,1)=fread(fid, 1, 'real*8')';
end

Is there a way to read this in without doing a loop? It would greatly speed up the read in. I just want three vectors with x,y,z. I just want to speed up the read in process. Thanks. Other suggestions welcomed.

score 3 · Accepted Answer · answered Dec 09 '12 at 06:15

3

I do not have a machine with Matlab and I don't have your file to test either but I think coordinates = fread (fid, [3, Ntot], 'real*8') should work fine.

answered Dec 09 '12 at 06:15

carandraug

12,938
1
26
38

@Griff the slowness of your code also came from the fact that you were using a parfor in a bad situation. You should have used a simple for loop here. Read [this answer](http://stackoverflow.com/a/3174364/1609556) aside your use of a for loop, you were also using a parfor loop for something that was too simple. If a loop – carandraug Dec 09 '12 at 07:14

score 0 · Answer 2 · answered Dec 09 '12 at 06:02

0

Maybe fread is the function you are looking for.

answered Dec 09 '12 at 06:02

rputikar

1,523
13
14

score 0 · Answer 3 · answered Dec 09 '12 at 06:15

You're right. Reading data in larger batches is usually a key part of speeding up file reads. Another part is pre-allocating the destination variable zeros, for example, a zeros call.

I would do something like this:

%Pre-allocate
x0 = zeros(Ntot,1);
y0 = zeros(Ntot,1);
z0 = zeros(Ntot,1);

%Define a desired batch size.  make this as large as you can, given available memory.
batchSize = 10000;

%Use while to step through file    
indexCurrent = 1;           %indexCurrent is the next element which will be read
while indexCurrent <= Ntot

    %At the end of the file, we may need to read less than batchSize
    currentBatch = min(batchSize,  Ntot-indexCurrent+1);

    %Load a batch of data
    tmpLoaded = fread(fid, currentBatch*3, 'read*8')';

    %Deal the fread data into the desired three variables
    x0(indexCurrent + (0:(currentBatch-1))) = tmpLoaded(1:3:end);
    y0(indexCurrent + (0:(currentBatch-1))) = tmpLoaded(2:3:end);
    z0(indexCurrent + (0:(currentBatch-1))) = tmpLoaded(3:3:end);

    %Update index variable
    indexCurrent = indexCurrent + batchSize;
end

Of course, make sure you test, as I have not. I'm always suspicious of off-by-one errors in this sort of work.

How to read binary file in one block rather than using a loop in matlab

3 Answers3