Fusion of Active and Passive Sensors for Dynamic 3D Capture

The outcome of this research project will be a robust and efficient depth sensing system. The system can be used in various tasks in computer vision and computer graphics including geometry reconstruction, robotics, human computer interaction, and mixed reality.Based on the sensor used, available depth sensing techniques can be classified into two categories: active and passive depth sensing methods. An active sensor emits energy toward targets to be investigated. Energy reflected from the targets is then detected and measured by the sensor. Passive sensors, on the other hand, do not have their own energy source and can be used only when the naturally occurring energy is available.The information provided by the active and passive sensors is largely complementary. Active sensors require the generation of a relatively large amount of energy to adequately illuminate targets, thus are generally of low resolution, sensitive to textured regions, vulnerable to motion blur, and have short measurement range. In contrast, passive sensors are of high resolution, in favor of textured regions, and have little motion blur and wide measurement range.We propose to synergistically combine passive and active sensors to create state-of-the-art depth sensing system with the following advantages:‧ Real-time high resolution depth sensingWe propose to enhance the spatial resolution of depth images captured by an active depth camera using a high resolution passive stereo camera. The new method promises an edgeaware, real-time and accurate solution to range data upsampling problem.‧ Fast motion depth sensingThe project next investigates into active depth camera motion deblurring problem. We propose to use the images captured by the passive stereo camera to estimate the point spread function in the active depth camera. The result of the second phase of the project will be a depth sensing system capable of capturing fast motion.‧ Robust depth sensingThe final phase of the project involves integrating the upsampled and deblurred depth images into an energy minimization framework, combining the advantages of active depth sensing and passive stereo. The output will be a depth sensing system that is robust to low texture and highly textured regions, and has wide measurement range.Some of the preliminary findings in this proposal were published in proceedings of IEEE International Workshop on Multimedia Signal Processing 2010 (MMSP’10), IEEE Conference on Computer Vision and Pattern Recognition 2010 and 2007 (CVPR’10 and 07) and IEEE Transactions on Pattern Analysis and Machine Intelligence 2009 (PAMI’09).