Out of Memory using bsxfun MATLAB

Question

I try to implement image compression using Burrows-Wheeler transform. Consider an 1D matrix from path scanning is:

p = [2 5 4 2 3 1 5];

and then apply the Burrows-Wheeler transform :

function output = bwtenc(p) 
n = numel(p);
x = zeros(length(p),1);
for i = 1:length(p)
    left_cyclic = mod(bsxfun(@plus, 1:n, (0:n-1).')-1, n) + 1;
    x = p(left_cyclic);
end
[lex ind] = sortrows(x);
output = lex(:,end);
output = uint8(output(:)');
end

And it works! But the problem is when i try to implement 1D matrix from Lena.bmp which the size is 512*512, Error message showing that bsxfun is out of memory. Anyone please help me.

Is the loop necessary? `x` seems to take the same values in all iterations — Luis Mendo, Sep 15 '14 at 22:37
Thanks for your revised. I have already delete the loop but still same problem. Error message : error using bsx fun, out of memory. Please help — yudha25, Sep 15 '14 at 22:41
Sure, the memory problem wasn't because of the loop. `bsxfun` is trying to create a 512^2 x 512^2 square matrix; that's the problem — Luis Mendo, Sep 15 '14 at 22:48
How could creating a 512x512 matrix be a problem? What system config are you on? What is size of `p`? — Divakar, Sep 15 '14 at 22:51
So do you have any suggestion for me? I use bsxfun to make left cyclic permutation. Maybe you have any code to replace it? I'm very appreciate your help — yudha25, Sep 15 '14 at 22:52
The size of p is 1 Dimensional matrix from 512*512 = 1*262144. I don't know why this error message showed. Do you have any suggestion for me? — yudha25, Sep 15 '14 at 22:57
@yudha25 Sorry, I'd like but I don't see how. If I understand correctly, your `p` is 512^2 x 1, so `left_cyclic` (or `x`) is 512^2 x 512^2. That's a large matrix. I don't see how to avoid creating it, since you need to sort its rows — Luis Mendo, Sep 16 '14 at 07:38

Divakar · Answer 1 · 2014-09-15T23:13:31.527

0

See if this works for you -

function output = bwtenc(p) 
np = numel(p);
[~,sorted_ind] = sort(p);
ind1 = mod((1:np)+np-2,np)+1;
output = p(ind1(sorted_ind));
output = uint8(output(:)');
end

edited Sep 15 '14 at 23:13

answered Sep 15 '14 at 23:07

Divakar

218,885
19
262
358

I'm very appreciate your help @Divakar. But that's not i want. I want the output is – yudha25 Sep 15 '14 at 23:52
p = [2 5 4 2 3 1 5; 5 4 2 3 1 5 2; 4 2 3 1 5 2 5; 2 3 1 5 2 5 4; 3 1 5 2 5 4 2; 1 5 2 5 4 2 3; 5 2 5 4 2 3 1]; – yudha25 Sep 15 '14 at 23:55
@yudha25 When I use `p = [2 5 4 2 3 1 5]` as input to your code, I get this - `[3 4 5 2 5 1 2]` instead. So, correct your code first? – Divakar Sep 16 '14 at 04:41
@yudha25 The output you want would be - `p(left_cyclic)` i.e. `p(mod(bsxfun(@plus, 1:n, (0:n-1).')-1, n) + 1)`. If your p is `1*262144`, you are looking to get an output `262144x262144` anyway. So, you MUST get out of memory error! But in any case, the output from your function doesn't match up with what you just said as the expected output in the comment above. Could you clarify on all of these issues? – Divakar Sep 16 '14 at 09:06
that's the problem. How can i avoid that memory error? – yudha25 Sep 16 '14 at 15:39
@yudha25 If you try to fill 2 liters of water into a 1 liter cup, you can't do much with it. Makes sense now? At my system, I can't hold `262144 x 262144` data, so if I were you I would go for cluster MATLAB (if I have access to cluster) or just try some other language. – Divakar Sep 16 '14 at 16:11
I understand now. But if we divide 1*262144 into small matrices and do that code, can I? – yudha25 Sep 16 '14 at 16:18
@yudha25 I asked you one question earlier - `"When I use p = [2 5 4 2 3 1 5] as input to your code, I get this - [3 4 5 2 5 1 2] instead. So, correct your code first?"` which was in response to your comment that the output must be the output must be - `p = [2 5 4 2 3 1 5; 5 4 2 3 1 5 2; 4 2 3 1 5 2 5; 2 3 1 5 2 5 4; 3 1 5 2 5 4 2; 1 5 2 5 4 2 3; 5 2 5 4 2 3 1]`. Explain that first. – Divakar Sep 16 '14 at 16:20
My answer above is the output of left cyclic permutation. And finally i take the last column of that left cyclic as result @Divakar – yudha25 Sep 16 '14 at 16:37
@yudha25 Okay so looking at it closely it appears that because of `sortrows`, you HAVE to process all of that data of `262144 x 262144` at once, which means your task is not possible with the existing single system MATLAB. Either go for cluster MATLAB or use some other software platform/language or look for another algorithm to reduce datasize. – Divakar Sep 17 '14 at 06:09

Out of Memory using bsxfun MATLAB

1 Answers1

Linked