Specularity and Shadow Interpolation via Robust Polynomial Texture Maps

Polynomial Texture Maps (PTM) [1] form an alternative method for apprehending surface colour and albedo that extends a simple model of image formation from the Lambertian variant of Photometric Stereo (PST) to more general reflectances. Here we consider solving such a model in a robust version, not to date attempted for PTM. But the main upshot of utilizing robust regression is in the identification of both shadows and specularities automatically, without the need for any thresholds, in a tripartite set of weights for pixels that are labelled as matte, shadow, or specularity. Original images are captured using a hemispherical set of lights, and pixel values across the lighting directions are then labelled as inliers, or outliers of two types. A per-pixel robust regression on luminance is carried out using Least Median of Squares, and automatically-identified outlier pixels are labelled as shadows if they are darker than matte and correspondingly, specular outliers are too bright. Inlier identification generates correct values for chromaticity and for surface albedo and thus matte luminance and colour. Then a robust version of PST, using only PTM inliers, improves estimates of normal vectors and albedo recovered. With specular pixel values over the lights in hand we model specularity using a radial basis function (RBF) regression, and non-specular pixel departures from matte using a second RBF set. Then for a new lighting direction, we can readily interpolate both specular content as well as shadows. Here, we are interested in using PTM as a vehicle to carry out interpolation of specularities and shadows. To the best of our knowledge robust methods have not to date been applied to PTM, and we use these to be able to accomplish interpolation. As well as using robust regression, this paper moreover shows how outliers can be classified as belonging to two types: either specular highlights, or selfor cast-shadows. Knowledge of inlier pixel values means that recovered surface albedo and chromaticity is robust, in the sense of ignoring outlier contributions and thus more accurately mapping surface reflectance and colour. Finally, knowledge of outlier labels means that we can independently model specularity and shadow. Then for a new lighting direction, we can generate pixel values interpolating known values of both; here we use a Radial Basis Function (RBF) interpolation model. In this paper we carry out robust regression on the luminance values, not on R,G,B separately. Since we generate the specularity ς in an interpolated image, separately from the remaining contribution σ , we can then produce a full-colour interpolant image using the luminance times matte chromaticity for the non-specular contribution, plus specular-luminance times the chromaticity for the specular colour. Results on re-creating the input images are shown to have excellent agreement with the originals, over a variety of input sequences. For shadow and specularity interpolation – generating images for nonobserved lighting directions – the method is shown to indeed generate sensible results. The main contributions in this paper are (1) application of robust regression to PTM; (2) separate modelling and thus better capturing of shadows and specularities; and (3) specular and shadow interpolation. Suppose we have acquired n images of a scene, taken from I = 1..n different lighting directions a I as in Fig. 1(a). Let each RGB image acquired be denoted ρ I . Suppose we make use of the luminance images instead, LI = ∑k=1 ρk. This reduction in dimensionality reduces the computational burden of robust regression and, since we mean to separate out the specularity from the matte component, we can re-insert colour later separately for matte colour and specular colour. Then a PTM model consists of a nonlinear regression from lighting to luminance via a vector of polynomial terms p , with p a function of lighting direction a , as follows:  p (a 1) p (a 2) ... p (a n)  c =  L1 L2 ... Ln  , or P c = L (1)