RainyScape : Unsupervised Rainy Scene Reconstruction using Decoupled Neural Rendering
Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review
Author(s)
Related Research Unit(s)
Detail(s)
Original language | English |
---|---|
Title of host publication | MM '24 |
Subtitle of host publication | Proceedings of the 32nd ACM International Conference on Multimedia |
Place of Publication | New York, NY, United States |
Publisher | Association for Computing Machinery |
Pages | 10920-10929 |
Number of pages | 10 |
ISBN (print) | 979-8-4007-0686-8 |
Publication status | Published - 28 Oct 2024 |
Link(s)
DOI | DOI |
---|---|
Document Link | |
Link to Scopus | https://www.scopus.com/record/display.uri?eid=2-s2.0-85204818747&origin=recordpage |
Permanent Link | https://scholars.cityu.edu.hk/en/publications/publication(647309ed-980f-49c9-8ec1-96b823ff60ff).html |
Abstract
We propose RainyScape, an unsupervised framework to reconstruct pristine scenes from a collection of multi-view rainy images. RainyScape consists of two main modules: a neural rendering module and a rain-prediction module that incorporates a predictor network and a learnable latent embedding that captures the rain characteristics of the scene. Specifically, leveraging the spectral bias property of neural networks, we first optimize the neural rendering pipeline to obtain a low-frequency scene representation. Subsequently, we jointly optimize the two modules, driven by the proposed adaptive direction-sensitive gradient-based reconstruction loss, which encourages the network to distinguish between scene details and rain streaks, facilitating the propagation of gradients to the relevant components. Extensive experiments on both the classic neural radiance field and the recently proposed 3D Gaussian splatting demonstrate the superiority of our method in effectively eliminating rain streaks and rendering clean images, achieving state-of-the-art performance. The constructed high-quality dataset, source code, and supplementary material are publicly available at https://github.com/lyuxianqiang/RainyScape.
Bibliographic Note
Research Unit(s) information for this publication is provided by the author(s) concerned.
Citation Format(s)
RainyScape: Unsupervised Rainy Scene Reconstruction using Decoupled Neural Rendering. / Lyu, Xianqiang; Liu, Hui; Hou, Junhui.
MM '24: Proceedings of the 32nd ACM International Conference on Multimedia. New York, NY, United States: Association for Computing Machinery, 2024. p. 10920-10929.
MM '24: Proceedings of the 32nd ACM International Conference on Multimedia. New York, NY, United States: Association for Computing Machinery, 2024. p. 10920-10929.
Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review