1

I'm trying to reconstruct position of chessboard from its stereo correspondences taken from calibrated stereo camera images. According to this answer, I went through whole process as

  1. Calibrate stereo camera
  2. Calculate projection matrices
  3. Triangulate undistorted points

I don't understand one step, that is WHY should I rectify stereo images during stereoCalibration using openCV's cv2::stereoRectify() or Matlab's rectifyStereoImages. Or maybe better: HOW should I get projection matrices - by stereo rectification or simply as 

P = K×[R|t]

from R, t and K gotten from stereoCalib / cameraCalib function.

As I understood the topic, stereo rectification is somehow important for making depth map, but as I've only ever worked with 3D estimation by triangulation, I am not sure if it is anyhow useful in this case. I've got a bit confused because every single stereo calibration demo I found contains this rectification step.

More complex answers are welcome as I think I figured the answer is simply calculate it by provided equation, but I would like to understand more what this rectification is about.

Dan
  • 77
  • 7
  • Stereo calibration tells you how to convert one point from one camera to a different point in the next one. Its the thing that allows you to **not** do 3D *estimation*, because you calibrated it. No need for any type of estimation. That way you have accurate 3D representation, not just a guess.This is fundamental if you want any type of accuracy (note how nowadays most of the systems have the cameras phisically attached to each other, to avoid estimation and calibration). – Ander Biguri Oct 29 '19 at 11:25
  • You always have only a guess in the world of nonlinear optimization. I would be really careful with words like _accurate_. – Dan Oct 29 '19 at 11:51
  • Besides that, I don't even claim that Stereo Calibration is an estimation in the post (to be clear with terminology). If the post itself is misleading, please tell me, I will try to make it more clear. – Dan Oct 29 '19 at 12:08
  • I think it has to do with the facts that doing math in epipolar geometry is easier, but its been too many years since I played with this – Ander Biguri Oct 29 '19 at 12:54

1 Answers1

2

If you already know which points correspond to each other on two images you are right, stereo rectification is not needed. You can just triangulate the points to get the 3D coordinate.

But if you are trying to estimate a depth map from a whole image you will need to solve the stereo correspondence problem, i.e. you will need to find for each point in one image the corresponding point in the second image.

By stereo rectifying you ensure that epipolar lines are horizontal and corresponding points only can be on the same horizontal pixel row in both images. This means your search space for correspondences is radically smaller and thus the problem becomes tractable.

Let me know if this answers your question or if you need a few more pointers into the right direction!

Malefitz
  • 417
  • 3
  • 7