Learning to Explore Saliency for Stereoscopic Videos Via Component-Based Interaction

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journal

View graph of relations

Author(s)

  • Xu Wang
  • Zhenhao Sun
  • Jianmin Jiang

Related Research Unit(s)

Detail(s)

Original languageEnglish
Pages (from-to)5722-5736
Journal / PublicationIEEE Transactions on Image Processing
Volume29
Publication statusPublished - 9 Apr 2020

Abstract

In this paper, we devise a saliency prediction model for stereoscopic videos that learns to explore saliency inspired by the component-based interactions including spatial, temporal, as well as depth cues. The model first takes advantage of specific structure of 3D residual network (3D-ResNet) to model the saliency driven by spatio-temporal coherence from consecutive frames. Subsequently, the saliency inferred by implicit-depth is automatically derived based on the displacement correlation between left and right views by leveraging a deep convolutional network (ConvNet). Finally, a component-wise refinement network is devised to produce final saliency maps over time by aggregating saliency distributions obtained from multiple components. In order to further facilitate research towards stereoscopic video saliency, we create a new dataset including 175 stereoscopic video sequences with diverse content, as well as their dense eye fixation annotations. Extensive experiments support that our proposed model can achieve superior performance compared to the state-of-the-art methods on all publicly available eye fixation datasets.

Research Area(s)

  • Visual saliency, stereoscopic video, deep learning