Compressed Domain Deep Video Super-Resolution

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review

View graph of relations

Author(s)

  • Wenhan Yang
  • Long Sun
  • Kangkang Hu

Related Research Unit(s)

Detail(s)

Original languageEnglish
Article number9509352
Pages (from-to)7156-7169
Journal / PublicationIEEE Transactions on Image Processing
Volume30
Online published9 Aug 2021
Publication statusPublished - 2021

Abstract

Real-world video processing algorithms are often faced with the great challenges of processing the compressed videos instead of pristine videos. Despite the tremendous successes achieved in deep-learning based video super-resolution (SR), much less work has been dedicated to the SR of compressed videos. Herein, we propose a novel approach for compressed domain deep video SR by jointly leveraging the coding priors and deep priors. By exploiting the diverse and ready-made spatial and temporal coding priors (e.g., partition maps and motion vectors) extracted directly from the video bitstream in an effortless way, the video SR in the compressed domain allows us to accurately reconstruct the high resolution video with high flexibility and substantially economized computational complexity. More specifically, to incorporate the spatial coding prior, the Guided Spatial Feature Transform (GSFT) layer is proposed to modulate features of the prior with the guidance of the video information, making the prior features more fine-grained and content-adaptive. To incorporate the temporal coding prior, a guided soft alignment scheme is designed to generate local attention off-sets to compensate for decoded motion vectors. Our soft alignment scheme combines the merits of explicit and implicit motion modeling methods, rendering the alignment of features more effective for SR in terms of the computational complexity and robustness to inaccurate motion fields. Furthermore, to fully make use of the deep priors, the multi-scale fused features are generated from a scale-wise convolution reconstruction network for final SR video reconstruction. To promote the compressed domain video SR research, we build an extensive Compressed Videos with Coding Prior (CVCP) dataset, including compressed videos of diverse content and various coding priors extracted from the bitstream. Extensive experimental results show the effectiveness of coding priors in compressed domain video SR.

Research Area(s)

  • coding prior, deep learning, soft alignment, super-resolution, Video compression

Citation Format(s)

Compressed Domain Deep Video Super-Resolution. / Chen, Peilin; Yang, Wenhan; Wang, Meng; Sun, Long; Hu, Kangkang; Wang, Shiqi.

In: IEEE Transactions on Image Processing, Vol. 30, 9509352, 2021, p. 7156-7169.

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review