Semantic-embedded Unsupervised Spectral Reconstruction from Single RGB Images in the Wild

Research output: Chapters, Conference Papers, Creative and Literary WorksRGC 32 - Refereed conference paper (with host publication)peer-review

12 Scopus Citations
View graph of relations

Related Research Unit(s)

Detail(s)

Original languageEnglish
Title of host publication2021 IEEE/CVF International Conference on Computer Vision ICCV 2021
Subtitle of host publicationProceedings
PublisherIEEE
Pages2259-2268
ISBN (electronic)978-1-6654-2812-5
ISBN (print)978-1-6654-2813-2
Publication statusPublished - Oct 2021

Publication series

NameInternational Conference on Computer Vision (ICCV)
ISSN (Print)1550-5499
ISSN (electronic)2380-7504

Conference

TitleIEEE International Conference on Computer Vision 2021
LocationVirtual
Period11 - 17 October 2021

Abstract

This paper investigates the problem of reconstructing hyperspectral (HS) images from single RGB images captured by commercial cameras, without using paired HS and RGB images during training. To tackle this challenge, we propose a new lightweight and end-to-end learning-based framework. Specifically, on the basis of the intrinsic imaging degradation model of RGB images from HS images, we progressively spread the differences between input RGB images and re-projected RGB images from recovered HS images via effective unsupervised camera spectral response function estimation. To enable the learning without paired ground-truth HS images as supervision, we adopt the adversarial learning manner and boost it with a simple yet effective L1 gradient clipping scheme. Besides, we embed the semantic information of input RGB images to locally regularize the unsupervised learning, which is expected to promote pixels with identical semantics to have consistent spectral signatures. In addition to conducting quantitative experiments over two widely-used datasets for HS image reconstruction from synthetic RGB images, we also evaluate our method by applying recovered HS images from real RGB images to HS-based visual tracking. Extensive results show that our method significantly outperforms state-of-the-art unsupervised methods and even exceeds the latest supervised method under some settings. The source code is public available at https://github.com/zbzhzhy/ Unsupervised-Spectral-Reconstruction.

Bibliographic Note

Research Unit(s) information for this publication is provided by the author(s) concerned.

Citation Format(s)

Semantic-embedded Unsupervised Spectral Reconstruction from Single RGB Images in the Wild. / Zhu, Zhiyu; Liu, Hui; Hou, Junhui et al.
2021 IEEE/CVF International Conference on Computer Vision ICCV 2021: Proceedings. IEEE, 2021. p. 2259-2268 (International Conference on Computer Vision (ICCV)).

Research output: Chapters, Conference Papers, Creative and Literary WorksRGC 32 - Refereed conference paper (with host publication)peer-review