MSU Technologies

A Novel Framework for Lidar-Camera Fusion

VALUE PROPOSITION

Motion-Capture, the essence of any autonomous vehicle application is the process of recording and following the movement of objects and people. The use of this 3D-based object detection enables better spatial planning and object avoidance than its 2D counterparts. 2D object detection systems often lack sufficient video resolution which has an adverse impact at longer ranges, this framework works well at both long and short ranges.

DESCRIPTION OF TECHNOLOGY

This technology named CLOCs (Camera-LiDAR Object Candidates) provides a better performing, low-complexity edge to 3D object detection by employing the use of both camera imaging and 3D LiDAR data. In this CLOC’s method, there is a fusion between these 2D images and 3D LiDAR data. By employing a late fusion framework, it has a significant advantage in training; single modality algorithms can be trained using their own sensor data. This technology can use any pair of pre-trained 2D and 3D detectors without requiring retraining.

BENEFITS

Versatility & Modularity: CLOCs uses any pair of pre-trained 2D and 3D detectors without requiring retraining, and hence, can be readily employed by any relevant already-optimized detection approaches.
Probabilistic-driven Learning-based Fusion: CLOCs is designed to exploit the geometric and semantic consistencies between 2D and 3D detections and automatically learns probabilistic dependencies from training data to perform fusion.
Speed and Memory: CLOCs is fast, leveraging sparse tensors with low memory footprint, which only adds less than 3ms latency for processing each frame of data.
Detection Performance: CLOCs improves single modality detectors, including state-of-the-art detectors, to achieve high performance levels.

APPLICATIONS