Spectral Resolution Enhancement of Hyperspectral Images via Sparse Representations

High-spectral resolution imaging provides critical insights into important computer vision tasks such as classification, tracking, and remote sensing. Modern Snapshot Spectral Imaging (SSI)systems directly acquire the entire 3D data-cube through the intelligent combination of spectral filters and detector elements. Partially because of the dramatic reduction in acquisition time, SSI systems exhibit limited spectral resolution, for example, by associating each pixel with a single spectral band in Spectrally Resolvable Detector Arrays. In this paper, we propose a novel machine learning technique aiming to enhance the spectral resolution of imaging systems by exploiting the mathematical framework of Sparse Representations (SR). Our formal approach proposes a systematic way to estimate a high-spectral resolution pixel from a measured low-spectral resolution version by appropriately identifying a sparse representation that can directly generate the highspectral resolution output. We enforce the sparsity constraint by learning a joint space coding dictionary from multiple low and high spectral resolution training data and we demonstrate that one can successfully reconstruct high-spectral resolution images from limited spectral resolution measurements. Introduction Over the last decade, the demand for designing imaging systems able to reveal the physical properties of the objects in a scene of interest, has grown tremendously. To that end, Hyperspectral Imaging has emerged as a powerful technology, able to capture and process a huge amount of data, including the spatial and spectral variations of an input scene. This type of data is crucial for multiple applications, such as remote sensing, precision agriculture, food processing, medical and biological research, etc. Despite the important merits of hyperspectral imaging systems in structure identification and remote sensing, HSI acquisition and processing also comes with multiple functional constraints. Slow acquisition time, limited spectral and spatial resolution, low dynamic range, and restricted field of view, are just a few of the limitations that hyperspectral sensors exhibit, and which require further investigation. The rapid evolution of the spectrally resolvable detector array systems that directly acquire the entire 3D data-cube through a clever combination of spectral filters and detector elements, has created an enormous excitement in the hyperspectral imaging community [1–5]. Although these systems acquire the entire spatial and spectral information directly from a single snapshot image, they unfortunately cause a reduction in spectral resolution by associating each detector/pixel with a single spectral band. The spectral resolution is a critical parameter for both visualization and subsequent procedures such as unmixing [6, 7], classification [8–10], and understanding of the variations of an input scene over time. Traditional hyperspectral resolution enhancement approaches focus mostly on the spatial resolution of HSI systems. In the remote sensing community, conventional techniques generate the high-spatial resolution scene by fusing a low spatial resolution hyperspectral image with a high spatial resolution panchromatic image, a procedure known as pan-sharpening [11]. Another class of techniques utilizes spatio-spectral relations to improve spatial resolution [12, 13]. Furthermore, over the past decade multiple techniques have been proposed that seek to enhance the spatial resolution of multispectral imagery by exploiting the Sparse Representations framework [13, 14]. More specifically, the authors in [14] propose a sparsity-based approach based on the assumption that corresponding pairs of high and low spatial resolution pixel curves share the same sparse codes with respect to appropriate resolution dictionaries. In order to improve the quality of their reconstruction, a maximum a priory algorithm is utilized. Contrary to the spatial resolution enhancement problem, in the spectral domain only a handful of techniques have been reported. The authors in [16] propose a spectral super-resolution approach applied on a generalization of the Coded Aperture Snapshot Spectral Imaging (CASSI) instrument, increasing simultaneously both the spectral and the spatial dimensions of hyperspectral scenes. Additionally, in [17] another spectral resolution enhancement is demonstrated, where the authors consider geographically co-located multispectral and hyperspectral oceanic water-color images and they enhance the limited multispectral measurements utilizing a sparsity based approach. First, they use a spectral mixing formulation and they define the measured spectrum for each pixel as the sum of the weighted material spectra. The desired high-spectral resolution spectra is expressed as a linear combination between a blurring matrix and the measured spectra. This problem is solved via a sparse-based technique. Our proposed algorithm aims to enhance the spectral dimension, i.e., the number of acquired spectral bands, providing richer and more thorough descriptions of a scene of interest. Instead of introducing hardware solutions for spectral-resolution enhancement, such as modifying the optics or the hyperspectral sensor characteristics, the proposed scheme adheres to a signal learning paradigm, offering convenient post-acquisition processing, able to extract a richer spectral information from a limited number of spectral bands. The proposed spectral super-resolution method is formulated as an inverse imaging algorithm that recovers highspectral information from low-spectral resolution data acquired Figure 1: Block diagram of the proposed scheme: Our algorithm takes as input a hypercube acquired with a limited number of spectral bands and constructs an estimate of the full spectrum of the scene. by the spectral detectors, by capitalizing on the Sparse Representations (SR) and the joint dictionary learning frameworks [18,19], effectively encoding the relationships between high and low spectral resolution “hyper-pixels”. Spectral Super-Resolution Using Sparsity This work proposes a novel scheme for synthesizing a highspectral resolution hypercube from a limited number of acquired spectral bands. More specifically, given a low-spectral resolution hyperspectral scene acquired with M spectral bands, our goal is to estimate the extended spectrum composed of N spectral bands, where N > M. In order to achieve this, we employ the Sparse Representations framework [18], which states that linear combinations between high-frequency signals can be accurately recovered from their corresponding low-frequency linear representations. The notion of sparsity has revolutionized modern signal processing and machine learning, and has lead to very impressive results in a variety of imaging problems, including superresolution, de-nighting etc. [20, 21]. Instead of observing directly the high-spectral resolution components, we work with double over-complete dictionaries, Dh for the high-spectral, and D` for the low-spectral resolution scenes. The sparse code of the low-spectral resolution part in terms of D`, will be combined with the high spectral resolution dictionary to generate the desired high-spectral resolution component. Formally, given a low-spectral resolution input hypercube S`, ”hyper-pixels” s` ∈ RM are extracted and mapped to the lowspectral resolution dictionary matrix D` ∈RM×P containing P examples. Subsequently, we seek to identify the sparse code vector w∈RP, with respect to the corresponding low-spectral resolution dictionary matrix. Recovery of the sparse code w is achieved by solving the following minimization problem: min w ||w||0 subject to ||s`−D`w||2 < ε, (1) where ε stands for the acceptable approximation error which is related to the added noise. This problem can be solved by a greedy strategy such as the Orthogonal Matching Pursuit algorithm [22]. Alternatively, one can replace the non-zero counting `0 pseudonorm by its convex surrogate `1-norm: ||w||1 = ∑i |wi|, and solve the corresponding problem given by: min w ||s`−D`w||2 +λ ||w||1, (2) where λ is a regularization parameter, a formulation known as the LASSO problem [23]. By considering the joint training of the low and high spectral resolution dictionaries, the objective is to identify the sparse code vector that can produce both the low and the high spectral resolution representations. Consequently, assuming that such an optimal sparse code w? is found by solving Eq. 2, we recover the high spectral resolution ”hyper-pixel” sh, by projecting w? to the high-spectral resolution dictionary, Dh, according to: sh = Dhw (3) The two main challenges for the proposed spectral resolution enhancement scheme is the sufficient sparsity measure for the sparse vector w, and the proper construction of the dictionary matrices D`, and Dh which will allow the sparsification of both low and higher spectral resolution data. In the following, an efficient scheme for multiple feature space dictionary learning is provided. Coupled Dictionary Construction Consider a set composed of high and low spectral resolution hypercubes. We assume that these scenes are realized by the same statistical process under different spectral resolution conditions, and as such, they share approximately the same sparse code with respect to their corresponding dictionaries, Dh and D`. A straightforward strategy to create these dictionaries is to randomly sample multiple correspondent “hyper-pixels” extracted from corresponding high and low spectral resolution training sets and use this random selection as the sparsifying dictionary. However, such a strategy is not able to guarantee that the same sparse code can be utilized among the two different representations. To overcome this limitation, we propose learning a compact dictionary from such pairs of high and low-spectral resolution data-cubes. Given a large set of training “hyper-pixels” extracted from multiple pairs of high and low spectral resolution hyperspectral scenes Sh and S`, our goal is to learn a joint dictionary D j, taking into account both representations. Consequently, the joint dictionary learning problem is formulated as: min D j ,X ||P−D jX||2 +λ ||X||1, s.t ||D j(:, i)||2 ≤ 1 (4) where D j = [ Dh D` ] ∈ R(M+N)×P, M+N denotes the concatenated number of spectral bands for both high and low-spectrum scenarios, P is the number of dictionary atoms, and P = [ Sh S` ]