2

I've used OpenCV to calibrate my camera from different views and I obtained intrinsics, rvec and tvec with a reprojection error of .03 px (I thus think the calibration is fine).

Now, given one view of my scene, I want to be able to click on a point and find its projection on the other views. To do so, I se the following functions:

void Camera::project(const vector<cv::Point2f> &pts_2d, vector<cv::Point3f> &pts_3d) {

     std::cout << "Start proj 2d -> 3d" << std::endl;

     cv::Mat pts_2d_homo;
     convertPointsToHomogeneous(pts_2d, pts_2d_homo);

     std::cout << "Cartesian to Homogeneous done!" << std::endl;

     // Project point to camera normalized coordinates    
     cv::Mat unproj;

     cv::transform(pts_2d_homo, unproj, intrinsics().inv());

     std::cout << "Point unprojected: " << unproj.at<cv::Point3f>(0) << std::endl;

     // Undo model view transform
     unproj -= transVec();

     cv::Mat rot;
     cv::Rodrigues(rotVec(), rot);
     cv::transform(unproj, unproj, rot.t());

     unproj *= 1.f/cv::norm(unproj);

     std::cout << "Model view undone: " << unproj.at<cv::Point3f>(0) << std::endl;

    for (int i = 0; i < unproj.rows; ++i) {
        std::cout << "Inside for :" << unproj.at<cv::Point3f>(i,0) << std::endl;
        pts_3d.push_back(unproj.at<cv::Point3f>(i,0));
   }
}

void Camera::project(const vector<cv::Point3f> &pts_3d, vector<cv::Point2f> &pts_2d) {
    cv::projectPoints(pts_3d, rotVec(), transVec(), intrinsics(), dist_coeffs(), pts_2d);

}

Now I have mixed feelings about what I get as an output. When I draw the point projected on each view, they all correspond BUT no matter where I clicked at first in the "canonical view", the projected point is always the same.

Jérémy Riviere
  • 173
  • 1
  • 10
  • In order to click a point in an image and have it projected in other views, you would need to know the depth of the particular point you are clicking. The line `unproj *= 1.f/cv::norm(unproj);` seems to be trying to do that, but without knowing the depth of the 3D point it cannot work. – BConic Sep 21 '15 at 09:01
  • My understanding was that the first step of applying the inverse intrinsic matrix would generate a ray in eye space for my first view which I can then express in the terms of world coordinates by applying the inverse of the extrinsic. Then by rotating that ray into another view's coordinate and projecting to the image plane I should get the corresponding point on the image view. But I guess without depth, inverting the translation component does not make sense. Is that correct? – Jérémy Riviere Sep 21 '15 at 09:24
  • Applying the inverse intrinsic matrix generates a ray in *normalized* eye space. That is where you need the point depth to convert from normalized eye space to metric eye space. If the depths are not available, another solution would be to click the point in *two* views, so that you can triangulate it before projecting it in other views. – BConic Sep 21 '15 at 10:08
  • Thanks for that precision AldurDisciple. I've got to find a way to get a dense depth map from my three views then. – Jérémy Riviere Sep 21 '15 at 10:26
  • Since my scene is essentially planar, I should be able to generate rays in normalized eye space, intersect those rays with my plane right? – Jérémy Riviere Sep 23 '15 at 18:15
  • Right. Although if your scene is planar, you could also just use an homography to warp pixel coordinates from an image to the other (one homography per pair of images). – BConic Sep 23 '15 at 18:48
  • I guess I must be missing something here. I generate my rays by inverting the intrinsics matrix. Then I undo the rotation on the direction of the rays and translation on their origin point (which is the same for each ray, ie camera position). After that I compute the intersection of each ray with the plane that goes through (0,0,0) with normal (0,0,1) and reproject to the image plane. Except the points lie outside the image plane ... – Jérémy Riviere Sep 23 '15 at 18:50
  • Your approach is totally legitimate, if the scene is planar and you know the equation of the plane with respect to the cameras. If you don't know the plane equation, another approach is to estimate an homography mapping points from an image to the other. Check out [this document](https://cseweb.ucsd.edu/classes/sp04/cse252b/notes/lec04/lec4.pdf) on this topic. – BConic Sep 23 '15 at 21:08
  • It took me a while and a few pulled hairs to figure out what was wrong, but I finally have projections that make sense. The problem is that I was inverting the intrinsics (check), inverting the rotation on the generated rays (check), inverting the translation of the rays origin (check) but forgot to undo the rotation on the rays origin. I'll post an answer when I have refactored my code. – Jérémy Riviere Sep 24 '15 at 11:18

0 Answers0