Learning-based High-quality Hyperspectral Images Reconstruction

Abstract

Hyperspectral (HS) images densely collect the rich electromagnetic spectrum information, which provide more accurate and faithful measurements towards the real-world scences/objects. Consequently, it has grown increasingly popular over the past ten years in various fields, such as military, industrial, and scientific arenas. However, due to the hardware limitation of existing imaging systems, there is an inevitable trade-off between the spectral and spatial resolution. For a specific optical system, it could only record the image with either high spatial resolution together with very limited spectral bands, e.g., the high resolution multispectral (HR-MS) images , or dense spectral bands with reduced spatial resolution, e.g., the low resolution HS (LR-HS) images. Thus, to tackle such challenging issues, this thesis investigates learning-based approaches for the reconstruction of high-quality HS images. Additionally, in consideration of different degradations of HS images, i.e., HR-MSI and LR-HS images, we adaptively build separate reconstruction framework to effectively recover HR-HS images from relating input patterns.

To begin with, we take the most intuitive route to resolving the reconstruction issue by merging an LR-HS images and an HR-MSI. The cross-modality distribution of the spatial and spectral information makes the problem challenging. Inspired by the classic wavelet decomposition-based image fusion, we propose a novel lightweight deep neural network-based framework, namely progressive zero-centric residual network (PZRes-Net), to address this problem efficiently and effectively. Specifically, PZRes-Net learns a high-resolution and zero-centric residual image, which contains high-frequency spatial details of the scene across all spectral bands, from both inputs in a progressive fashion along the spectral dimension. And the resulting residual image is then superimposed onto the up-sampled LR-HS images in a meanvalue invariant manner, leading to a coarse HR-HS images, which is further refined by exploring the coherence across all spectral bands simultaneously. To learn the residual image efficiently and effectively, we employ spectral-spatial separable convolution with dense connections. In addition, we propose zero-mean normalization implemented on the feature maps of each layer to realize the zero-mean characteristic of the residual image.

Moreover, we investigate the reconstruction problem from perspective of spatially super-resolving single LR-HS images. Particularly, we focus on how to embed the high-dimensional spatial-spectral information of HS images efficiently and effectively. Specifically, in contrast to existing methods adopting empirically-designed network modules, we formulate HS embedding as an approximation of the posterior distribution of a set of carefully-defined HS embedding events, including layerwise spatial-spectral feature extraction and network-level feature aggregation. Then, we incorporate the proposed feature embedding scheme into a source-consistent super-resolution framework that is physically-interpretable, producing PDE-Net, in which high-resolution HS images are iteratively refined from the residuals between input LR-HS images and pseudo-LR-HS images degenerated from reconstructed HR-HS images via probability-inspired HS embedding. Extensive experiments over three common benchmark datasets demonstrate that PDE-Net achieves superior performance over state-of-theart methods. Besides, the probabilistic characteristic of this kind of networks can provide the epistemic uncertainty of the network outputs, which may bring additional benefits when used for other HS image-based applications.

On the other hand, we further recover HS images from only single RGB images. To tackle such a severely ill-posed problem, we propose a physically-interpretable, compact, efficient, and end-to-end learning-based framework, namely AGD-Net. Precisely, by taking advantage of the imaging process, we first formulate the problem explicitly based on the classic gradient descent algorithm. Then, we design a lightweight neural network with a multi-stage architecture to mimic the formed amended gradient descent process, in which efficient convolution and novel spectral zero-mean normalization are proposed to effectively extract spatial-spectral features for regressing an initialization, a basic gradient, and an incremental gradient. Besides, based on the approximate low-rank property of HS images, we propose a novel rank loss to promote the similarity between the global structures of reconstructed and ground-truth HS images, which is optimized with our singular value weighting strategy during training. Moreover, AGD-Net, a single network after onetime training, is flexible to handle the reconstruction with various spectral response functions. The proposed framework achieve desirable performance . Note that, it is non-trivial to collect a large number of such paired data via specially designed devices. Thus, to enable the learning without paired ground-truth HS images as supervision, we adopt the adversarial learning manner and boost it with a simple yet effective L1 gradient clipping scheme. Besides, we embed the semantic information of input RGB images to locally regularize the unsupervised learning, which is expected to promote pixels with identical semantics to have consistent spectral signatures. Extensive experiments over both real and synthetic benchmark datasets demonstrate that our approaches outperforms state-of-the-art methods to a significant extent in terms of different quantitative metrics and visual quality.

Date of Award	12 Jul 2023
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Junhui HOU (Supervisor)