Perception-Oriented U-Shaped Transformer Network for 360-Degree No-Reference Image Quality Assessment

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review

View graph of relations

Author(s)

  • Mingliang Zhou
  • Lei Chen
  • Qin Mao
  • Heqiang Wang
  • Huayan Pu
  • Jun Luo
  • Tao Xiang
  • Bin Fang

Related Research Unit(s)

Detail(s)

Original languageEnglish
Number of pages10
Journal / PublicationIEEE Transactions on Broadcasting
Online published4 Jan 2023
Publication statusOnline published - 4 Jan 2023

Abstract

Generally, 360-degree images have absolute senses of reality and three-dimensionality, providing a wide range of immersive interactions. Due to the novel rendering and display technology of 360-degree images, they have more complex perceptual characteristics than other images. It is challenging to perform comprehensive image quality assessment (IQA) learning by simply stacking multichannel neural network architectures for pre/postprocessing, compression, and rendering tasks. To thoroughly learn the global and local features in 360-degree images, reduce the complexity of multichannel neural network models and simplify the training process, this paper proposes a joint architecture with user perception and an efficient transformer dedicated to 360-degree no-reference (NR) IQA. The input of the proposed method is a 360-degree cube map projection (CMP) image. Furthermore, the proposed 360-degree NRIQA method includes a saliency map-based non-overlapping self-attention selection module and a U-shaped transformer (U-former)-based feature extraction module to account for perceptual region importance and projection distortion. The transformer-based architecture and the weighted average technique are jointly utilized for predicting local perceptual quality. Experimental results obtained on widely used databases show that the proposed model outperforms other state-of-the-art methods in NR 360-degree image quality evaluation cases. Furthermore, a cross-database evaluation and an ablation study also demonstrate the inherent robustness and generalization ability of the proposed model.

Research Area(s)

  • Image quality assessment, no-reference image quality assessment, 360-degree image, U-shaped transformer, OMNIDIRECTIONAL IMAGE, SALIENCY, CNN