How to append in .mat file row or columnwise cellmatrix

Question

I am running a simulation where i generate huge 2d sparse matrices and hence i use FIND function to only store nonzero values with their indices.

Now for each iteration of for loop i generate such matrix and because they are all of different length I use cell-array to store these configurations. But for large simulations even squeezed-off format of cell-array crosses its limits of memory and hence i want to write these cell-array while running the code i.e. for each iteration append a new element into existing mat file.

for e.g.

for n=1:10  
A=rand(5);  
[i,j,vals]=find(A);  
data={[i,j,vals]};  
save('data','data','-append');  
end

Here my final goal is to have a mat file where number of elements in "data" are 10. but i because of memory issues i cant save it outside the for loop i want to generate data{n} and append it in columnwise growing fashion. eventually giving me data{10}.

I tried to use MATFILE but it gives me error that it doesnt work with {} hence not working with cell arrays.

Thank you, Nitin

That is a really nice question. Although I do not know how to solve your question I would just want to ask you why would you intend to do that. If you have that problem at saving the variable you will also have it at loading in the future. Therefore you should avoid that. Why not saving it as Luis Mendo suggests? — ASantosRibeiro, Jun 06 '14 at 13:52
With MATFILE function(if that works with cell-arrays) if I get it write part of .mat file then i can also read part of a file without loading it. hence all i want is MATFILE to work with cell-arrays if it does. — nitin, Jun 06 '14 at 14:45
I'm curious why you are not using MATLAB's native sparse format instead of rolling your own. — nhowe, Jun 06 '14 at 15:48

score 1 · Accepted Answer · answered Jun 06 '14 at 16:36

You can use matfile with cells as long as you are not trying to index into the actual cells. Remember that cells are still arrays, so you can access each cell using array indexing. For example:

>> x = {'this', 'is', 'an', 'example'};
>> x{4}

ans =

example

>> x(4)

ans = 

    'example'

The following just initializes the data. Make sure to save it with the '-v7.3' tag, so that it supports efficient partial saving and loading.

data = {};
save('data.mat', 'data', '-v7.3');

Now you can access your data with the matfile

mf = matfile('data.mat', 'Writable', true);
for n=1:10  
   A=rand(5);  
   [i,j,vals]=find(A);  
   data={[i,j,vals]};
   mf.data(end+1, 1) = data;
end

Reference: matfile documentation

thanks a lot. This is roughly what i was looking for. – nitin Jun 06 '14 at 20:45 — nitin, Jun 06 '14 at 20:45

score 0 · Answer 2 · answered Jun 06 '14 at 13:46

You can't append data to an existing variable with save. You need different variables:

clear all
filename = 'data.mat';
save(filename) %// empty file, for now. We'll append variables within the loop
for n = 1:10  
    A = rand(5);  
    [i,j,vals] = find(A);  
    varname = ['data' num2str(n)]; %// varname is 'data1', 'data2' etc
    assignin('base',varname,[i,j,vals]); %// create that variable
    save(filename, varname, '-append') %// append it to file
end

I thought of it but problem is I will have like millions of variables in my .mat file. Thanks. — nitin, Jun 06 '14 at 14:43

How to append in .mat file row or columnwise cellmatrix

2 Answers2

Linked