0

I am trying to project a 2D image from a 3D point cloud. Herein, the 3D point cloud is constructed using RGB and Depth images whose FOV is 120 degree. Once the 3D point cloud is constructed, I wanted to reproject a 2D image from it using a FOV 80 degree. The resolution of the images used for building 3D point cloud and the reprojected 2D image are same [width: 1820, height: 940].

I calculated the intrinsic matrix by following the below steps

The camera model used here is PinHole.

Step 1: Calculating focal lengths fx and fy

hfov = fov / 360. * 2. * np.pi
fx = img_w / (2. * np.tan(hfov / 2.))

vfov = 2. * np.arctan(np.tan(hfov / 2) * img_h / img_w)
fy = img_h / (2. * np.tan(vfov / 2.))

* fx and fy are in pixel length

Step 2: Calculating image centers u0, v0

u0, v0 = img_w / 2, img_h / 2

Step 3: Form the matrix

[[fx, 0, u0, 0],
 [0, fy, v0, 0],
 [0, 0, 1, 0],
 [0, 0, 0, 1]]

I used this intrinsic matrix and the extrinsic matrix (calculated from x,y,z, roll, ptich and yaw) to reproject the 2D image with 80 degree FOV using Open3D api from 3D point cloud which was constructed using 120 degree FOV.

When I follow the above steps, I ended up with a 2D image which is zoomed in a lot than expected. However, if I reproject the 2D image with 120 degree FOV then the resulting image is almost close to the expectation (some extra regions are projected but camera to plane distance is perfect).

You can find the sample output below

Reprojected image with FOV 80 degree from the point cloud (ego vehicle will be missing because it is not part of the RGB-D images) enter image description here

Reference image (taken with same FOV (80 degree), resolution, and extrinsic parameters) enter image description here

I am very certain that there is no issue with the extrinsic matrix calculation and the reprojection part because it is well tested and verified. The zoom in effect comes from the intrinsic matrix.

I am assuming that I am missing or doing something wrong while calculating fx and fy. Perhaps, there may be a need of adjustment factor while dealing with different FOV's or some inconsistencies in the units.

Any help here would be appreciated.

Vinod prime
  • 81
  • 1
  • 13
  • see if you can provide a [mre]. text is not debuggable. – Christoph Rackwitz Oct 05 '22 at 09:40
  • How did you decide that the "zoom" is stronger than expected? Can you provide a bird eye sketch of your scene with drawn 120° and 80° and the acutual visible points in the "zoomed" image? – Micka Oct 05 '22 at 12:27
  • This is a image generated from CARLA simulator. I verified by capturing the viewpoint directly with the same FOV and extrinsic matrix. In short, I have a reference image for the reprojected 2D image. – Vinod prime Oct 05 '22 at 14:05
  • @Micka I have added a same output for your reference. – Vinod prime Oct 06 '22 at 01:19
  • I don't know CARLA good enough. Where is the reference image coming from, is it also capturing the point cloud or a reference scene? Are you sure the point cloud is generated correctly? If both are 80°, what kind of didferes are there (scene or camera model)? – Micka Oct 06 '22 at 05:26
  • @Micka Reference image is coming directly from the camera. Yes, point cloud is generated correctly but only thing to note here is fx and fy we calculate are in pixels not in standard units like mm, m or cm. The difference lies in the scene. – Vinod prime Oct 06 '22 at 05:32
  • What's the difference between "the camera" and the other camera where you are projecting to? How do you know that the reference camera is 80°? – Micka Oct 06 '22 at 05:37
  • Because it is defined by me. In CARLA, I can define the FOV, reosultion and extrinsic parameter I needed in order to capture a scene. The reprojected image is obtained from the 3D point cloud which is constructed using 120 FOV from surround view cameras. – Vinod prime Oct 06 '22 at 06:54
  • Can't you get/read the intrinsics from the CARLA camera? – Micka Oct 06 '22 at 22:57
  • Does this help? https://github.com/carla-simulator/carla/discussions/5189 – Micka Oct 06 '22 at 23:02
  • Nope CARLA don't have any explicit api to generate intrinsic matrix. We need to form the matrix by using image height, width and FOV. – Vinod prime Oct 07 '22 at 02:13

0 Answers0