-1

I obtain depth & reflectance maps from Lidar (2D images) and I have also camera images (2D images). Image have the same size.

I want to use CNN to perform object detection using both images. It is a sort of "fusion CNN"

How am I suppose to do it? Did I am suppose to use a pre-train model? But the is no pre-train model using lidar images..

Which is the best CNN algorithm to do it? ie for performing fusion of modalities for object detection

Thanks you in advance

Doxcos44
  • 135
  • 2
  • 12

1 Answers1

0

Did I am suppose to use a pre-train model?

Yes you should, unless you are super confident that you can find a working model directly by urself.

But the is no pre-train model using lidar image

First I`m pretty sure there are LIDAR based network .e.g

L Caltagirone , LIDAR-Camera Fusion for Road Detection Using Fully Convolutional ... arxiv, ‎2018

Second, even if there is no open source implementation for direct LIDAR-based, You can always convert the LIDAR to the depth image. For Depth image based CNN, there are hundreds of implementation for segmentation and detection.

How am I suppose to do it?

First, you can place them side by side parallel, for RGB and depth/LIDAR 3d pointcloud. Feed them separately

Second, you can also combine them by merging the input to 4D tensor and transfer the initial weight to the single model. At last perform transfer learning in your given dataset.

best CNN algorithm?

Totally depends on your task and hardware. Do you need best in processing speed or best in accuracy? Define your "best", please.

ALso Are you using it for autonomous car or for in-house nurse care system? different CNN system customizes the weight for different purposes.

Generally, for real-time multiple object detection using a cheap PC e.g DJI manifold, I would suggest Yolo-tiny

Dr Yuan Shenghai
  • 1,849
  • 1
  • 6
  • 19