A New High Resolution Depth Map Estimation System Using Stereo Vision and Kinect Depth Sensing

Depth map estimation is an active and long standing problem in image/video processing and computer vision. Conventional depth estimation algorithms which rely on stereo/multi-view vision or depth sensing devices alone are limited by complicated scenes or imperfections of the depth sensing devices. On the other hand, the depth maps obtained from the stereo/multi-view vision and depth sensing devices are de facto complementary to each other. This motivates us to develop in this paper a new system for high resolution and high quality depth estimation by joint fusion of stereo and Kinect data. We modeled the observations using Markov random field (MRF) and formulated the fusion problem as a maximum a posteriori probability (MAP) estimation problem. The reliability and the probability density functions for describing the observations from the two devices are also derived. The MAP problem is solved using a multiscale belief propagation (BP) algorithm. To suppress possible estimation noise, the depth map estimated is further refined by color image guided depth matting and a 2D polynomial regression (LPR)-based filtering. Experimental results and numerical comparisons show that our system can provide high quality and high resolution depth maps, thanks to the complementary strengths of both stereo vision and Kinect depth sensors.