Photometric Stereo for General Reflectance and Lighting in Partial Fulfillment of the Requirements for the Degree of Doctor of Information Science and Technology

Understanding the 3D shape information is a fundamental problem in computer vision. Among various shape estimation technology, photometric stereo is highlighted for its ability to produce detailed surface normal direction at its image resolution. It takes a set of images captured under varying illumination and a fixed viewpoint as input. Traditional photometric stereo assumes the Lambertian reflectance and distant lighting. These assumptions are seldom satisfied in a practical scenario. This dissertation generalizes the assumptions for photometric stereo, specifically on reflectance and lighting, towards creating a practical surface normal sensor. The proposed approaches serve as fundamental support to the design of future cameras which are able to record and measure 3D shapes for various applications like culture heritage archive, digital museum, virtual reality, 3D scene understanding, and so on. The first general reflectance solution exploits the reflectance monotonicity for estimating elevation angles of surface normal given the azimuth angles to fully determine the surface normal. With an assumption that the reflectance includes at least one lobe that is a monotonic function of the angle between the surface normal and half-vector (bisector of lighting and viewing directions), we prove that elevation angle can be uniquely determined when the surface is observed under varying directional lights densely and uniformly distributed over the hemisphere. The second general reflectance solution is built upon a newly developed reflectance model. We notice that if the high-frequency reflectance can be neglected, the lowfrequency component of general reflectance can simply be represented using loworder polynomials. Based on this observation, we propose a compact bi-polynomial reflectance model to describe the general isotropic reflectance precisely in the lowfrequency domain. We apply our reflectance model to radiometric image analysis problems of estimating reflectance and shape given recorded scene radiance, namely reflectometry and photometric stereo, for surfaces with general reflectance. Both of the proposed solutions for general reflectance have been evaluated by using a densely measured reflectance database containing one hundred different materials and various types of real-world data. These evaluations cover a diversity of common materials in our daily life; hence the experiments prove that our approaches are valid for a broad class of reflectance and useful for various practical scenarios. The third solution is about general lighting. We present a photometric stereomethod that works with general/unknown lightings and uncontrolled sensors using a coarse shape that is given. We show that the coarse shape information, or a shape prior, serves to solve two difficult issues: removing shape-light ambiguity in unknown natural lightings, and disregarding uncontrolled sensor gains and responses. Our method is well-suited to work with a low-cost RGBD camera, whose radiometric characteristics