0

Context: I'm running a job (consisting of 1 to 50 tasks) on a cluster, waiting for it to finish, and then pulling the results using:

outputArgs = getAllOutputArguements(myJob);

The Problem: The getAllOutputArguements() function is returning a gigantic cell array, the size of which can vary from [1x1] cells to [1x50] cells. Each cell of outputArgs contains another array of cells, which can range from [1x1] to [1x15], and in each of those cells is an array of doubles which can vary from [1x106] elements to [500x106] elements. These ranges have the potential to get larger in the future. This generates an "Out of Memory" error, because the size of outputArgs exceeds the allocated memory for MATLAB.

The Question: What I need to do is save each top-level cell as a file. However, as far as I can tell, getAllOutputArguements is an atomic/indivisible operation for MATLAB. Does anyone know a way to assign each cell to a file, without first placing the output of getAllOutputArguements into a variable?

Caveat: I know that this problem could be solved by increasing my data limit, but the application I am building is going to be distributed, and I need a more elegant, less hardware-dependent solution.

Justin Fletcher
  • 2,319
  • 2
  • 17
  • 32
  • just out of curiosity, have you tried 'load'ing the variable/element you want from the job? – oligilo May 03 '14 at 17:14
  • I've tried various combinations of save and load. None have worked yet. – Justin Fletcher May 03 '14 at 22:53
  • I was getting the out of memory error when I was doing something roughly similar (http://stackoverflow.com/questions/23227520/downtime-between-calls-to-workers-in-matlab); there I realized that I hit the error because I keep the job open for many iterations so I passed the var's from them to the client and deleted them after every call. But if this is not your problem, it might be useful to spread the tasks over different workers, if you're allowed. – oligilo May 03 '14 at 23:48
  • Nope, in this case there's a memory bottleneck at the return value local assignment. – Justin Fletcher May 31 '14 at 03:14

0 Answers0