Convolutional Neural Network Based Synthesized View Quality Enhancement for 3D Video Coding

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journal

7 Scopus Citations
View graph of relations

Related Research Unit(s)


Original languageEnglish
Pages (from-to)5365-5377
Journal / PublicationIEEE Transactions on Image Processing
Issue number11
Online published20 Jul 2018
Publication statusPublished - Nov 2018


The quality of synthesized view plays an important role in the three dimensional (3D) video system. In this paper, to further improve the coding efficiency, a convolutional neural network (CNN) based synthesized view quality enhancement method for 3D High Efficiency Video Coding (HEVC) is proposed. Firstly, the distortion elimination in synthesized view is formulated as an image restoration task with the aim to reconstruct the latent distortion free synthesized image. Secondly, the learned CNN models are incorporated into 3D HEVC codec to improve the view synthesis performance for both view synthesis optimization (VSO) and the final synthesized view, where the geometric and compression distortions are considered according to the specific characteristics of synthesized view. Thirdly, a new Lagrange multiplier in the rate-distortion (RD) cost function is derived to adapt the CNN based VSO process to embrace a better 3D video coding performance. Extensive experimental results show that the proposed scheme can efficiently eliminate the artifacts in the synthesized image, and reduce 25.9% and 11.7% bit rate in terms of peak-signal-to-noise ratio (PSNR) and structural similarity (SSIM) index, which significantly outperforms the state-of-theart methods.

Research Area(s)

  • 3D high efficiency video coding, Convolutional neural network, depth coding, Distortion, Encoding, Estimation, Image coding, Lagrange multiplier, Optimization, Three-dimensional displays, Video coding, view synthesis