Image is warped after stereo rectification

Question

I'm currently implementing the stereovision with OpenCV. Now I'm using the Stereo_Calib sample to remove the distortion en rectify the image. Removing the distortion works fine.

enter image description here

But when I apply rectification, the image is very warped.

enter image description here

This is the code to rectify the images. The parameters rmap are calculated in the same way as in the Stereo_calib example (see here)

void StereoCalibration::StereoRectify(Mat &imageLeft, Mat &imageRight)
{
Mat imLeft, imRight;

remap(imageLeft, imLeft,DistLeft.rmap[0], DistLeft.rmap[1], CV_INTER_CUBIC);            
remap(imageRight,imRight, DistRight.rmap[0], DistRight.rmap[1], CV_INTER_CUBIC);

imageLeft = imLeft;
imageRight = imRight;   
}

the cause of such results is bad calibration images. anyone having such issues *must* present their calibration images for scrutiny. — Christoph Rackwitz, Aug 28 '22 at 15:41

score 2 · Answer 1 · edited May 23 '17 at 12:30

I realise this question is a few years old however, I have recently had a similar issue. Building on morynicz answer about "bad chessboard" patterns to calibrate stereo images, I found that even with a slight deformation in your chessboard pattern, for example that it isn't flat, can produce large warping in the stereo image pair on rectification. The algorithms in OpenCV, for instance, assume a flat chessboard pattern is being presented such that any physical deformation in that pattern will be wrongly attributed to distortions in the camera optics (or in the relative orientations of the two camera sensors). The algorithms will then try really hard to remove this false distortion leading to very warped images.

To avoid this problem, were possible, use a tablet (or other electronic screen) to display the chessboard pattern as it is then guaranteed to be flat.

Additionally, you should check that the images you are using to calibrate the stereo pair are in focus and have no motion blur or image tearing.

If using OpenCV to do the rectification do some experimentation with the flags used in the stereoCalibrate function as this may lead to a more "optimised" rectification for your particular application.

score 1 · Answer 2 · edited Aug 28 '22 at 14:41

1

For anyone looking for help on this, I was dealing with very large scale resolution images and was getting very low reprojection error rate with good calibration images. I was getting very warped stereo pairs after rectification and a really bad depth map.

One thing to try is if your images are warped you might need to down-sample them.

Another thing to try is to combine the flags in stereoCalibrate instead of just choosing one.

Something like this worked for me :

cv2.stereoCalibrate(
    object_points, image_points_left,image_points_right,
    camera_matrix_left,dist_left,
    camera_matrix_right, dist_right,
    (5472,3648),None,None,None,None,
    cv2.CALIB_FIX_ASPECT_RATIO + cv2.CALIB_ZERO_TANGENT_DIST + cv2.CALIB_USE_INTRINSIC_GUESS + cv2.CALIB_SAME_FOCAL_LENGTH + cv2.CALIB_RATIONAL_MODEL,criteria
)

edited Aug 28 '22 at 14:41

Tyler2P

2,324
26
22
31

answered Aug 22 '22 at 20:50

devman3211

63
1
11

if the undistorted images were warped like in the question, then your calibration images were *not good*. reprojection error only tells you when the data is junk. good reprojection error does NOT mean the data is good. one typical mistake is to fail to cover a significant portion of the view with the grid pattern. – Christoph Rackwitz Aug 28 '22 at 15:40
So even with bad calibration, these stereo flags can still rectify and correct a proper stereo vision pair to create the depth map? I was using a 20 mp camera and the checkerboard pictures were done by an intern but we covered most of the "3D space" of the camera with different positions, rotations, and skews to about 40 pictures each. I was dealing with a very small space so small checkerboard was required( made out of opal at a machine shop). They seem to be very sensitive as to how they attempt to rectify - is this a result of bad calibration? I was still able to generate a great depth map – devman3211 Aug 28 '22 at 16:58
each image must contain a pattern with significant cover over the image. otherwise it's useless. anything with less than 10% area covered is marginal, if not junk. -- don't pass such tasks off to an intern. they know even less about this stuff, and they tend to *not question* any instructions or opinions you might have. or at least make sure they wrote a thesis on the math or something. https://calib.io/blogs/knowledge-base/calibration-best-practices – Christoph Rackwitz Aug 28 '22 at 17:35
Interesting thanks. So our checkerboard should be big enough to cover most of the frame when doing any of the orientations? How about the size of the squares/# of squares then? Do more image points make the calibration better? If we fit more squares in by changing the size of the squares while keeping the size of the chessboard the same The smaller the checkerboard squares, the harder it is to detect but when dealing with super high-res cameras, this would ensure better calibration or make no difference? – devman3211 Aug 28 '22 at 17:40
OpenCV's checkerboard detection requires all nodes to be visible. that makes them a bad choice (until I get some free time to fix that...) because you get few or no points at all in the corners of your camera view, where they're *most needed*. for now, go with "charuco", but even those have issues. resolution is irrelevant to _mistakes_ in calibration. more points only matter if they actually cover the entire image. lots of points right in the middle are pointless. and "diminishing returns". a billion points is silly. my gut says that 1k-10k points is more than enough. – Christoph Rackwitz Aug 28 '22 at 17:42

score 0 · Answer 3 · edited Feb 22 '20 at 20:44

I started working on opencv stereo image calibration and rectification recently and I was getting similar images. Although it is true to make sure the board is straight and it is true that we need to take multiple images on the corners and in the middle of the camera at different x,y,z and skew positions, what did the trick for me was the flags in stereoCalibrate. I used all the flags specified in the opencv docs except for INTRINSIC_GUESS and it started very nice undistorted and rectified images.

score 0 · Answer 4 · answered Dec 04 '11 at 14:32

0

I had the same problem, and I think that the issue was bad chessboard used to calibration or mixing up the maps.

answered Dec 04 '11 at 14:32

morynicz

2,322
2
20
34

Image is warped after stereo rectification

4 Answers4