It's guaranteed that R has a zero eigenvalue, and the solution you want is a multiple of the eigenvector corresponding to that eigenvalue. Let's create some matrix R first:
>> R = triu(rand(3, 3));
>> R(3, 3) = 0;
>> R
R =
0.8147 0.9134 0.2785
0 0.6324 0.5469
0 0 0
Now let's get the eigenvalues and eigenvectors:
>> [V, E] = eig(R)
V =
1.0000 -0.9806 0.4289
0 0.1958 -0.5909
0 0 0.6833
E =
0.8147 0 0
0 0.6324 0
0 0 0
The eigenvectors are the diagonal elements of E
>> E = diag(E);
>> index = find(abs(E) < 1e-16); %// NB don't use find(E==0) because of fp problems...
Now pull out the correct eigenvector
>> v = V(:, index);
and make sure its final element is equal to 1
>> v = v / v(end)
v =
0.6277
-0.8648
1.0000
You can check that this is the solution you want
>> R * v
ans =
0
0
0