Projecting a 2D point into 3D space using camera calibration parameters in OpenCV

Question

I get my camera parameters using calibrateCamera() and now I have cameraMatrix, distCoeffs, rotationMatrix, transformMatrix.
With these matrices I can build the Projection Matrix and convert 3D object points in the space into 2D image points.
Some thing like this:

But what I want is the reverse of this projection. I want to convert these 2D points back into 3D space. I know I'll lost some information, but all of my original points were in a same plan.
Please help me to build a similar matrix by using camera parameters for this convertion.

Answer 1

From a set of projected 2D point you cannot get back the original 3D points, but the rays that join the 3D points and their projections in the image plane. So, you lose the depth of the 3D points; it is to say, you know the orientation but you don't have the distance from the camera to the 3D points.

You will have to make up the depth of the 3D points. Their planar condition allows you to make some constraints in their relative positions, but it is not enough to retrieve their original depth.

For example, you can set the depth of 3 non-collinear points to create a plane in the 3D space. The depth of the other 2D points will be given by the intersection of those rays with this new plane.

If you know the normal vector to the plane that originated the 3D points, you could do something similar just by setting the depth of a single 2D point and computing the others accordingly. Only if you have, in addition, the distance from that plane to the origin (to the camera), you can retrieve the real depth of your 3D points.