Append data to a saved variable using matfile()

Question

I have some random 2D data fuffa that I save to a file fuffalo:

fuffa=rand(10,10);
save('fuffalo','fuffa', '-v7.3')

I then go through a loop that generates other fuffa data that I want to append (in the third dimension) to the saved variable. To do this, I tried to apply this suggestion:

m1 = matfile('fuffalo.mat', 'Writable', true);

for ii=1:3
   fuffa2=rand(10,10);    
   m1.fuffa(1:10,1:10,end+1)=fuffa2;
end

However, at ii=2 I get the following error:

Variable 'fuffa' has 2 dimensions in the file, this does not match the 3 dimensions in the indexing subscripts.

How can I convince MATLAB to append in the third dimension?

Can you append in the first dimension? `fuffa = rand(1,10,10)` — Gelliant, Sep 26 '17 at 08:05
Because matlab automatically removes trailing singleton dimensions after the first two. — Gelliant, Sep 26 '17 at 09:23
not (only) at `ii=2`, you should be getting the error for `ii=1` as well — Sardar Usama, Sep 26 '17 at 10:03
The first two dimensions. Not entries. So a [20 x 1] vector stays a [20 x 1] vector. But a [20 x 20 x 1] matrix becomes a [20 x 20] matrix. The third, singleton, dimension is removed. It's one of the quirks of matlab. — Gelliant, Sep 26 '17 at 11:32

Wolfie · Accepted Answer · 2017-09-26T15:14:46.223

Because you are accessing the file on disk, not a workspace variable, you may be encountering issues with expanding the number of dimensions. You would not have this issue dealing with variables which are stored in-memory (like if you used load instead of matfile).

To avoid this, the best approach would be pre-allocation. I'm going to assume though that this is a simplification of your actual problem, and you need to be able to do such 3D extensions to a potentially 2D array.

In this case, just use cat to concatenate in the 3rd dimension:

fuffa=rand(10,10);
save('fuffalo','fuffa', '-v7.3')
m1 = matfile('fuffalo.mat', 'Writable', true);
for ii=1:3
    fuffa2=rand(10,10);
    % Concatenating in the 3rd dimension, avoiding used 'end' which 
    % assumes that dimension already exists
    m1.fuffa=cat(3,m1.fuffa,fuffa2);
end
% m1.fuffa <10x10x4 double>

Note that by doing this, you are bringing the entirety of your .mat data into memory for the concatenation, defeating the point of matfile(). However, your previous method would face the same problem, since in the docs we see:

Using the end keyword as part of an index causes MATLAB to load the entire variable into memory.

As stated before, you're probably better off with pre-allocation!

The file is an HDF5 container. The data in it has fixed dimensions, but you could just as well first store a 3D matrix `10x10x2` and overwrite the second fuffa. It then is specified in the container as 3D and you should be able to append to it. — Gelliant, Sep 26 '17 at 11:29
The `matfile()` function allows the user to access data directly from the file on disk rather than in memory. "Workspace" was, again, probably the wrong choice of words by me. See the [docs](http://uk.mathworks.com/help/matlab/ref/matfile.html) for more, but basically this is faster than using `load` for large data sets, because the data is never fully loaded into memory. So a `MAT-file` object is in the workspace, but this doesn't contain the data, just references it. — Wolfie, Sep 26 '17 at 14:32

Append data to a saved variable using matfile()

1 Answers1