4

I'm using open cv in C++ in multi-view scene with two cameras. I have the intrinsic and extrinsic parameters for both cameras.

I would like to map a (X,Y) point in View 1 to the same point in the second View. I'm am slightly unsure how I should use the intrinsic and extrinsic matrices in order to convert the points to a 3D world and finally end up with the new 2D point in view 2.

Luke Zammit
  • 175
  • 2
  • 13
  • 1
    afaik you can only do one of these: if you know intrinsics and extrinsics of both cameras and know which pixel correspond you can compute 3d position of the point, or if 3d pos of point is known you can compute 2d image pos in camera where intr and extr params are known. however if you know homography you can compute 2d pos of pixel in 2nd cam even if you dont know any camera intrinsics/extr – Micka Mar 30 '15 at 17:33
  • @Micka thank you for your reply, i've just found the time to this out again. I've managed to extracted the Homograph 3x3 matrix using the two different views. However when I did: [x',y'] = H*[x,y] to get the points in the second views. The points are not well aligned.. :/ Any further ideas? – Luke Zammit Apr 02 '15 at 12:41
  • 1
    homographies are only true for 2 situations: 1. if both camera views have the same camera center point. 2. if the positions in 3D are all lying on a plane. different points can only the approximated by a homography transformation and the more their 3D points differ from a plane ()the plane you've used to compute the homography), the bigger the error will be. – Micka Apr 02 '15 at 13:20
  • @Micka Hmm from what i'm understanding if i have the Fundamental Matrix i can do use the fact that x'T . F . x = 0 Any ideas if I can get the Fundamental matrix from the intrinsic or extrinsic parameters please? – Luke Zammit Apr 03 '15 at 17:39
  • 1
    did you find this already? http://stackoverflow.com/questions/24783915/compute-fundamental-matrix-without-point-correspondences – Micka Apr 03 '15 at 19:13
  • 1
    Thanks @Micka for all your help, that seems to be what i require :) – Luke Zammit Apr 04 '15 at 13:47

1 Answers1

4

It is (normally) not possible to take a 2D coordinate in one image and map it into another 2D coordinate without some additional information.

The main problem is that a single point in the left image will map to a line in the right image (an epipolar line). There are an infinite number of possible corresponding locations because depth is a free parameter. Secondly it's entirely possible that the point doesn't exist in the right image i.e. it's occluded. Finally it may be difficult to determine exactly which point is the right correspondence, e.g. if there is no texture in the scene or if it contains lots of repeating features.

Although the fundamental matrix (which you get out of cv::StereoCalibrate anyway) gives you a constraint between points in each camera: x'Fx = 0, for a given x' there will be a whole family of x's which will satisfy the equation.

Some possible solutions are as follows:

  1. You know the 3D location of a 2D point in one image. Provided that 3D point is in a common coordinate system, you just use cv::projectPoints with the calibration parameters of the other camera you want to project into.

  2. You do some sparse feature detection and matching using something like SIFT or ORB. Then you can calculate a homography to map the points from one image to the other. This makes a few assumptions about things being planes. If you Google panorama homography, there are plenty of lecture slides detailing this.

  3. You calibrate your cameras, perform an epipolar rectification (cv::StereoRectify, cv::initUndistortRectifyMap, cv::remap) and then run them through a stereo matcher. The output is a disparity map which gives you exactly what you want: a per-pixel mapping from one camera to the other. That is, left[y,x] = right[y, x+disparity_map[y,x]].

(1) is by far the easiest, but it's unlikely you have that information already. (2) is often doable and might be suitable, and as another commenter pointed out will be poor where the planarity assumption fails. (3) is the general (ideal) solution, but has its own drawbacks and relies on the images being amenable to dense matching.

Josh
  • 2,658
  • 31
  • 36