MATLAB 3D sparse reconstruction issues. Somehow I can't get the final scatter plot of the points to work

Question

I'm trying to reconstruct the shape of a sail. I'm using the 3D sparse reconstruction method. I'm using two cameras with which I took two pictures. I managed to do the calibration of such cameras too. In the pictures it is possible to see the checkerboard and the code I wrote detects it properly.

Now, since my pictures are black and white and the quality of the cameras is quite low, I cannot use the detectFeatures method properly. Problems arise when I'm trying to use matchFeatures. To overcome this problem I decided to use instead a cpselect command. By doing so I can manually click on the features. The matching between points from the two views seems now to be correct. When I carry on with the code and try to reconstruct the 3D plot I get points all over the place. It seems deformed. The plot clearly does not represent the sail and I don't know why.

The code follows.

Thank you in advance

 % % Load precomputed camera parameters
 load IP_CalibrationCarlos.mat                              %Calibration        feature
 %
 I1 = imread('/Users/riccardocamin/Documents/MATLAB/Frames/Scan1.1.jpg');
 I2 = imread('/Users/riccardocamin/Documents/MATLAB/Frames/Scan2.1.jpg');
 %
 [I1, newOrigin1] = undistortImage(I1, cameraParameters, 'OutputView', 'full');
 [I2, newOrigin2] = undistortImage(I2, cameraParameters, 'OutputView', 'full');
 %
 I1 = imcrop(I1, [80 10 1040 1300]);                        %Necessary so    images have same size
 I2 = imcrop(I2, [0 10 1067 1300]);
 %
 squareSize = 82;                                     % checkerboard square     size in millimeters
 %
 [imagePoints, boardSize, pairsUsed] =   detectCheckerboardPoints(rgb2gray(I1), rgb2gray(I2));
 [refPoints1, boardSize] = detectCheckerboardPoints(rgb2gray(I1));
 [refPoints2, boardSize] = detectCheckerboardPoints(rgb2gray(I2)); 
 %
 % % Translate detected points back into the original image coordinates
 refPoints1 = bsxfun(@plus, refPoints1, newOrigin1);
 refPoints2 = bsxfun(@plus, refPoints2, newOrigin2);
 % 
 worldPoints = generateCheckerboardPoints(boardSize, squareSize);
 %
 [R1, t1] = extrinsics(refPoints1, worldPoints, cameraParameters);  %R = r t = translation
 [R2, t2] = extrinsics(refPoints2, worldPoints, cameraParameters);
 %
 % % Calculate camera matrices using the |cameraMatrix| function.
 cameraMatrix1 = cameraMatrix(cameraParameters, R1, t1);
 cameraMatrix2 = cameraMatrix(cameraParameters, R2, t2);
 %
 cpselect(I1, I2); % Save them as 'matchedPoints1'and 'matchedPoints2'
 %
 indexPairs = matchFeatures(matchedPoints1, matchedPoints2);
 % Visualize correspondences
 figure;
 showMatchedFeatures(I1, I2, matchedPoints1, matchedPoints2);
 title('Matched Features');
 %
 [points3D] = triangulate(matchedPoints1, matchedPoints2, ...
 cameraMatrix1, cameraMatrix2);
 %
 x = -points3D(:,1);
 y = -points3D(:,2);
 z = -points3D(:,3);
 figure
 scatter3(x,y,z, 25);
 xlabel('X');
 ylabel('Y');
 zlabel('Z');

Answer 1

The first problem you have is that you are cropping the images. Once you do that, all your coordinates are off. You do not need to do that here, because the images do not need to be the same size.

The second question is how precisely did you select the matching points? From the picture you have posted, it seems that your matches can be off by a few pixels, which can result in a large reconstruction error. Can you try finding the centroids of the spots on the sail using regionprops?

Also, are the two cameras stationary relative to each other? If so, then you may be better off calibrating the stereo pair, and doing the dense reconstruction as in this example. In that case, the two images would have to be of the same size.