Content-aware Warping for View Synthesis
Research output: Journal Publications and Reviews › RGC 21 - Publication in refereed journal › peer-review
Author(s)
Related Research Unit(s)
Detail(s)
Original language | English |
---|---|
Pages (from-to) | 9486-9503 |
Journal / Publication | IEEE Transactions on Pattern Analysis and Machine Intelligence |
Volume | 45 |
Issue number | 8 |
Online published | 6 Feb 2023 |
Publication status | Published - Aug 2023 |
Link(s)
Abstract
Existing image-based rendering methods usually adopt depth-based image warping operation to synthesize novel views. In this paper, we reason the essential limitations of the traditional warping operation to be the limited neighborhood and only distance-based interpolation weights. To this end, we propose content-aware warping, which adaptively learns the interpolation weights for pixels of a relatively large neighborhood from their contextual information via a lightweight neural network. Based on this learnable warping module, we propose a new end-to-end learning-based framework for novel view synthesis from a set of input source views, in which two additional modules, namely confidence-based blending and feature-assistant spatial refinement, are naturally proposed to handle the occlusion issue and capture the spatial correlation among pixels of the synthesized view, respectively. Besides, we also propose a weight-smoothness loss term to regularize the network. Experimental results on light field datasets with wide baselines and multi-view datasets show that the proposed method significantly outperforms state-of-the-art methods both quantitatively and visually. The source code will be publicly available at https://github.com/MantangGuo/CW4VS. © 2023 IEEE.
Research Area(s)
- View synthesis, light field, deep learning, image warping, depth/disparity
Bibliographic Note
Research Unit(s) information for this publication is provided by the author(s) concerned.
Citation Format(s)
Content-aware Warping for View Synthesis. / Guo, Mantang; Hou, Junhui; Jin, Jing et al.
In: IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 45, No. 8, 08.2023, p. 9486-9503.
In: IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 45, No. 8, 08.2023, p. 9486-9503.
Research output: Journal Publications and Reviews › RGC 21 - Publication in refereed journal › peer-review