On Sampling of Image-based Rendering Data

Image-based rendering (IBR) generates novel views from images instead of 3D models. It can be considered as a process of sampling the light rays in the space and interpolating the ones in novel views. The sampling of IBR is a high-dimensional sampling problem, and is very challenging. This thesis focuses on answering two questions related to IBR sampling, namely how many images are needed for IBR, and if such number is limited, where should we capture them. There are three major contributions in this dissertation. First, we give a complete analysis on uniform sampling of IBR data. By introducing the surface plenoptic function, we are able to analyze the Fourier spectrum of non-Lambertian and occluded scenes. Given the spectrum, we also apply the generalized sampling theorem on the IBR data, which results in better rendering quality than rectangular sampling for complex scenes. Such uniform sampling analysis provides general guidelines on how the images in IBR should be taken. For instance, it shows that non-Lambertian and occluded scenes often require higher sampling rate. Second, we propose a very general sampling framework named freeform sampling. Freeform sampling has three categories: incremental sampling, decremental sampling and rearranged sampling. When the to-be-reconstructed function values are unknown, freeform sampling becomes active sampling. Algorithms of active sampling are developed for image-based rendering and show better results than the traditional uniform sampling approach. Third, we present a self-reconfigurable camera array that we developed, which features a very efficient algorithm for real-time rendering and the ability of automatically reconfiguring the cameras to improve the rendering quality. Both are based on active sampling. Our camera array is able to render dynamic scenes interactively at high quality. To our best knowledge, it is the first camera array in literature that can reconfigure the camera positions automatically.