Efficient Spectral Imaging based on Imaging Systems with Scene Adaptation Using Tunable Color Pixels

Conventional spectral imaging systems use a set of predetermined filters to capture multi-band images. Liquid crystal tunable filters (LCTF) and active illumination allow reconfiguration of spectral sensitivities but these techniques have shortcomings such as latency due to multiple captures and the fact that the same filtering or illumination is applied to the whole frame of the image. There are emerging device technologies that allow independent adjustment of the filtering for each region or even at a pixel level of the imaging frame. The operation of such imaging systems is controlled by adapting to the scene based on scene analysis. Experiments were run by simulating a spectral imaging system which adjusts pixel sensitivities based on color information from the scene. As a result this new system exhibits superior performance compared to traditional spectral imaging systems in terms of color accuracy and imaging capture efficiency. Introduction Traditionally, spectral imaging has relied on the use of predetermined set of filters that are mechanically or electronically adjusted to capture image bands with different spectral properties [1]. Spectral imaging has been confined to some niche high-end applications such as remote sensing [2] and artwork analysis and archiving [3]. The reason for not having consumer level spectral imaging products yet are due to several factors such as cost, bulkiness of the imaging system and lack of a compelling application. Some of the shortcomings of conventional spectral imaging systems are the need to increase the number of captured signals to increase spectral resolution. Moreover, spectral imaging systems are inherently by design very inefficient not just because of the tremendous redundancy in spectral information, but also because spectral imaging systems typically capture pre-determined channels regardless if there are meaningful information in the captured band. Spectral imaging is also conceptually a sub-category of computational imaging since it extends the capabilities of digital imaging by encoding and computing wavelength of light instead of trichromatic signals. However, spectral imaging has not exploited exhaustively the property of imaging capture re-configurability that is typical in computational imaging systems. One reconfigurable system is the Agile Spectrum Imaging [4] that is based on an adjustable computer-controlled optical system using a diffraction grating to disperse rays into different colors combined with an electronically controlled mask in the optical path to control spectrum. Another spectral imaging with reconfigurable approach is the spectral vision system that uses an optical set-up with a liquid-crystal spatial light modulator to implement color filters [5]. Such implementations show the possibilities of computational spectral imaging. Tunable Imaging Sensors It is possible to make a leap in terms of miniaturization of reconfigurable spectral imaging devices by exploring imaging sensors with tunable spectral sensitivities. There are primarily two types of tunable sensors in the literature: sensors whose sensitivities can be tuned in the imaging sensor level, and sensors which have tunable color filter arrays. Recently, researchers from Politecnico di Milano in Italy proposed a new type of imaging sensor whose sensitivities can be tuned by changing the sensitivities of the sensors themselves [6-8]. This new type of imaging sensor is called Transverse Field Detector (TFD). The TFD takes advantage of the light absorption properties of Silicon. A key optical property of Silicon is that it absorbs different wavelengths of light depending at different depths. Thus, the lower parts of the sensor will absorb longer wavelengths compared to the upper parts of the sensor. By connecting surface electrodes, which produce transverse electric fields throughout the substrate that take advantage of the drift property of electrons, each electrode will then attract electrons coming from different depths in the Silicon. Since electrons coming from different depths in the Silicon are excited by different wavelengths of light, the electrodes can effectively capture the response from different wavelengths of light. By tuning these electrodes, which modify the drift properties of electrons, different absorption spectrums can be obtained. It has been shown that such a sensor could be effective not only for white balance adjustment [9] but also for reconfiguring an imaging sensor for illumination level [10]. A second type of tunable sensor takes the form of tunable color filter arrays as described in [11]. Instead of tuning the sensitivities of the imaging sensor, the absorption spectrum of color filters are modulated instead. Though each pixel element can only record one channel, as in a classical imaging sensor, the spectral sensitivities for each one of these elements can be adjusted, as in the TFD. Since the TFD offers more advantages, we will be discussing using a theoretical device similar to the TFD (in the fact that every pixel has multiple channel read-outs) for the remainder of this study. Another related work is by Sajadi et. al [12], who proposed an image capture apparatus using switchable primaries by employing shiftable layers of color filter arrays. While this system cannot be tuned pixel-by-pixel, it is a type of adaptive imaging system, which modifies its sensor characteristics based on the scene. Efficient adaptive spectral imaging In [13], Langfelder et al. showed that by using the same TFD imaging sensor but by utilizing a non-symmetric electric biasing on the TFD, at a cost of a reduced fill factor due to extra read-out circuitry, it is possible to increase the number of captured channel 332 ©2011 Society for Imaging Science and Technology from 3 to 5. By redesigning the device it is possible to obtain even more channels by increasing the size of the pixel. This new functionality of the TFD eliminates the necessity of a color filter array and therefore reduces the overall complexity of the system. By using a tunable spectral imaging sensor, it is possible to build a reconfigurable spectral imaging system that adapts to the content of the scene, increasing capture efficiency. Specific spectral bands are more appropriate for certain reflectances of the scene. For example, if a region of the scene is predominantly red, then it is more appropriate to have pixel sensitivities absorb more red light. Thus, depending on the reflectance of the scene in various regions, the sensor can be tuned to be optimized for the specific reflectance in those regions. This work performs preliminary simulations of a theoretical imaging system that has spatially tuning spectral sensitivity, where the tuning is based on adaptation to the color content of the capturing scene by performing image analysis of a captured preview. We performed experiments on simulations of tunable sensors versus conventional sensors for multispectral imaging and compared S-CIELAB distances, spectral root mean square (RMS) error as well as metamerism indices for a range of most commonly used illuminants. Moreover, due to the nature of such tunable imaging sensor, each pixel site can capture data for multiple color channels, eliminating the need for demosaicing on the final image, allowing for higher resolution multispectral images to be captured. The final result is a tunable sensor system which allows for improved performance over traditional multispectral imaging systems Spectral estimation method Spectral estimation consists of an inverse problem estimating the full reflectance of the scene at every single position, given input from multiple channels per pixel. Numerous spectral reflectance techniques exist. The most representative methods are outlined in [14]. A few popular spectral estimation methods include the pseudo-inverse method, eigenvector analysis with least squares, and modified-discrete sine transformation. The spectral estimation method used for this study will focus on using the pseudo-inverse method, which produces a linear transformation from the input channels, to the full spectrum of light, by applying the pseudo-inverse operator. Preview Image Analysis and Filter Tuning Different pixel sensitivities are optimal for different radiances. For example, when the radiance is predominantly red, it is more useful to have more sensitivity curves capture the longer wavelengths, since most of the information is there. Sampling this curve more finely in the areas that contain more information will result in improved reflectance reconstruction. In practice tunable sensors such as the TFD cannot be arbitrarily tuned due to device constraints. Therefore, instead of simulating a completely tunable system we adopt an approach that considers a finite collection of sets of spectral sensitivities. Each set of spectral sensitivities will be referred to as a filter mode. Thus, the set of spectral sensitivities that are biased toward red regions of the scene would be an example of a filter mode. Figure 1: Data flow for the method used in the simulations. One possible way to bias spectral sensitivities is by applying weights that shifts the sensitivities either towards short or long wavelengths. We adopted seven different filter modes in this study since we empirically found out that seven filter modes would cover most relevant colors. Default mode corresponds to sensitivities that may be found typically in existing multispectral cameras today by equally sampling the visible spectrum, which show no bias towards specific wavelengths. Red mode corresponds to sensitivities that are more densely sampled in longer wavelengths. Blue mode corresponds to sensitivities that are more densely sampled in shorter wavelengths. Figure 1 is a flow diagram for explaining a method of image capture of a scene in which spectral selectivity is adjusted on a region-by-region basis, for imaging sensors with tunable spectral properties, so as to increase spectral differentiation for spectral content in the scene. A default capture parameter (spatial electronic mask) is applied to an imaging assembly having a spectral response which is tunable in accordance with the capture parameter. The spatial electronic mask determines the spectral sensitivities of the sensor for each region. The initial electronic mask could be dictated by the default filter mode. A preview image of a scene is captured and the sample image is analyzed. The optimal filter mode to use is determined based on the scene, on a region-by-region basis. See Figure 2 for an illustration of different filter modes. Additional filter modes are similarly biased towards different areas of the spectrum. From this information, a spectral mask is constructed for each filter mode. The spectral masks are applied as capture parameters to the imaging assembly by adjusting biasing voltages to each pixel location. These biasing voltages could be determined by look-uptable. Finally, a spectral image of the scene is captured and stored according to the tuned settings. Imaging capture with tunable filters