1

EDIT:

What I have: camera intrinsics, extrinsic from calibration, 2D image and depth map

What I need: 2D virtual view image

I am trying to generate a novel view(right view) for Depth Image Based Rendering. Reason for this is that only the left image and depth map are available at the receiver which has to reconstruct the right view (see image).

I want to know if these steps will give me the desired result or what I should be doing instead,

First, by using Camera Calibration toolbox for MATLAB by CalTech, the intrinsic, extrinsic matrices can be obtained.

Then, they can be mapped to 3D world points using calibration parameters by this method "http://nicolas.burrus.name/index.php/Research/KinectCalibration#tocLink2"

Now, I want to back project this to a new image plane(right view). The right view is simply a translation of left and no rotation because of the setup. How do I do this reconstruction?

Also, Can I estimate R and T from MATLAB stereo calibration tool and transform every point in original left view to right view using P2 = R*P1+T, P1 and P2 are image points of 3D world point P in the respective planes.

Any idea and help are highly appreciated, would rephrase/add details if the question is not clear.

Paolo Ferraiuoli
  • 122
  • 12
Priyamvadha Krishnakumar
  • 33
  • 1
  • 7

1 Answers1

1

(Theoretic answer*)

You have to define what R and T means. If I understand, is the Roto-translation of your (main) left camera. If you can map a point P (like your P1 or P2) in 3D space, the correspondance with a point m (I not call it p to avoid confusion) in your left camera is (unless you use a different convention (pseudocode)

m = K[R|t]*P

which

P1 = (X,Y,Z,1)
m  = (u',v',w)

but you want 2D coordinates so the coordinates in your left camera are:

u = u'/w 
v = v'/w

if you already roto-translated P1 into P2 (not very useful) is equal to (pseudocode)

                  1 0 0 0
m = K[I|0]*P = K*[0 1 0 0] * P2
                  0 0 1 0

Assume this is the theoretical relationship with a 3D point P with his 2D point in an image m, you may think to have you right camera in a different position. If there is only translation with respect to left camera, the right camera is translated of T2 with respect to the left camera and roto-translated of R/T+T2 with respect to the centre of the world. So the m' proiected point in your right camera should be (assuming that the cameras are equal means the have the same intrinsics K)

m' = K[R|T+T2]*P = K[I|T2]*P2 I is the identity matrix.

If you want to transform m directly into m' withot using 3D points you have to implement epipolar geometry.

  • If cameras are different with different K, if the calibration of R and T has not the same standard of calibration of K, this equation may not work. If calibration is not well-done, it could work but with errors.
marcoresk
  • 1,837
  • 2
  • 17
  • 29
  • Thank you for your explanation! – Priyamvadha Krishnakumar Sep 20 '16 at 11:53
  • Does the translation vector parameters represent translation along X,Y,Z axes? – Priyamvadha Krishnakumar Sep 28 '16 at 08:56
  • @PriyamvadhaKrishnakumar yes. Unless you have some different convention, is the translation of the "centre" of the camera from the origin of the world reference system – marcoresk Sep 28 '16 at 14:40
  • Thank you @marcoresk . When I do warping using roto-translation described in your explanation, how do I identify holes(pixels which are visible only in new view)? As I transform only those pixel coordinates which are present in left image. – Priyamvadha Krishnakumar Sep 29 '16 at 08:53
  • @PriyamvadhaKrishnakumar I think that you can only check if a point is projected "out of bounds", i.e. projected in pixel (1500,900) in an image of 1280x800 pixels. This is a point visible only in the first image, not in the second. The opposite case, I presume, will result in some pixel which remain black since they have not a value assigned. These are your holes I think. – marcoresk Sep 30 '16 at 13:07
  • Thank you, it worked! Can you tell me how do I find depth of right view pixels after transforming image using formula suggested? – Priyamvadha Krishnakumar Oct 06 '16 at 09:01
  • @PriyamvadhaKrishnakumar I'm a little confused by your question... unless your goal is represented by [this question](http://stackoverflow.com/questions/39898247/calculate-depth-map-of-right-image). If it is so, I think I can answer there but my answer will not fill your holes (and it starts from consideration on what you already have in this discussion) – marcoresk Oct 07 '16 at 15:01
  • Sorry for the confusion, I am not looking for hole filling code. My goal is represented by that question! Please answer! – Priyamvadha Krishnakumar Oct 09 '16 at 15:31
  • @PriyamvadhaKrishnakumar I answered. And sorry for my bad english, what I meant with "filling holes" wass exactly what you wrote in the question about black holes in the image. – marcoresk Oct 10 '16 at 09:17