2

I am trying to render 3D point cloud from the depth data which I saved from opengl framebuffer. Basically, I took different depth samples from different n viewpoints (which are already known) for the rendered model centered at (0, 0, 0). I successfully saved the depth maps but now I want to extract x, y, z coordinated from these depth maps. For this, I am back projecting point from image to world. To get world coordinates I use the following equation P = K_inv [R|t]_inv * p. to calculate the world coordinates.

To calculate the image intrinsics matrix I used information from the opengl camera matrix, glm::perspective(fov, aspect, near_plane, far_plane). The intrinsic matrix K is calculated as

enter image description here where enter image description here

If I transform the coordinates in camera origin (i.e., no extrinsic transformation [R|t]), I get a 3D model for a single Image. To fuse multiple depths maps, I also need extrinsic transformation which I am calculating as from the OpenGL lookat matrix glm::lookat(eye=n_viewpoint_coorinates, center=(0, 0, 0), up=(0, 1, 0)). The extrisnics matrix is calculated as below (ref: http://ksimek.github.io/2012/08/22/extrinsic/

enter image description here

But when I fuse two depth images they are misaligned. I think the extrinsic matrix is not correct. I also tried to use glm::lookat matrix directly but that does not work as well. The fused model snapshot is shown below

enter image description here

Can someone suggest, what is wrong with my approach. Is it the extrinsic matrix that is wrong (which I am damn sure of)?

genpfault
  • 51,148
  • 11
  • 85
  • 139
Rajat Sharma
  • 47
  • 1
  • 15
  • 3
    Your formulas don't make sense. `K` is not an equivalent of the OpenGL projection matrix you used for rendering, and when you use it inverted, it will not restore linear depth values from the hyperbolic Z values you got from the depth buffer, `[R|t]_inv` doesn't even exist as `R` is a 3x3 matrix, and if you add a translation column `t` you get a 3x4 matrix which doesn't have an inverse. Your multiplication order also doesn't make sense. – derhass Jan 09 '21 at 14:19
  • Before the transformation, I made all matrices to 4x4 so that I can do inverse and multiplication. – Rajat Sharma Jan 14 '21 at 09:38

1 Answers1

0

Finally, I managed to solve this by myself. Instead of doing transformation inside the OpenGL, I did the transformation outside of the OpenGL. Basically, I kept the camera constant and at some distance from the model and did rotation transformation on the model, and then finally render the model without lookat matrix (or just 4x4 identity matrix). I don't know why using lookat matrix does not gave me the result or maybe it is due something I was missing. To backproject the model into world coordinates I would just take the inverse of the exact transformation I did initially before feeding the model to OpenGL.

Rajat Sharma
  • 47
  • 1
  • 15
  • Hi, could you expand on your explanation please? I am working on something very similar but I am a bit confused. Intrinsic matrix: how did you compute the coordinates of the principal point from the projection matrix? I see that you can get f from the window dimensions and the fov, but do you then neglect the principal point (assumed [0,0]) and the skew (assumed 0)? Extrinsic matrix: are you suggesting that R above (from http://ksimek.github.io/2012/08/22/extrinsic/) isn't right? I see some info to go from openCV to openGL, but less the other way around, so any help is welcome. Thanks – antoinef Apr 25 '21 at 23:47