Patch Based Video Summarization With Block Sparse Representation

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review

13 Scopus Citations
View graph of relations

Author(s)

  • Shaohui Mei
  • Mingyang Ma
  • Shuai Wan
  • Zhiyong Wang
  • David Dagan Feng

Detail(s)

Original languageEnglish
Article number9069290
Pages (from-to)732-747
Journal / PublicationIEEE Transactions on Multimedia
Volume23
Online published16 Apr 2020
Publication statusPublished - 2021

Abstract

In recent years, sparse representation has been successfully utilized for video summarization (VS). However, most of the sparse representation based VS methods characterize each video frame with global features. As a result, some important local details could be neglected by global features, which may compromise the performance of summarization. In this paper, we propose to partition each video frame into a number of patches and characterize each patch with global features. Instead of concatenating the features of each patch and utilizing conventional sparse representation, we formulate the VS problem with such video frame representation as block sparse representation by considering each video frame as a block containing a number of patches. By taking the reconstruction constraint into account, we devise a simultaneous version of block-based OMP (Orthogonal Matching Pursuit) algorithm, namely SBOMP, to solve the proposed model. The proposed model is further extended to a neighborhood based model which considers temporally adjacent frames as a super block. This is one of the first sparse representation based VS methods taking both spatial and temporal contexts into account with blocks. Experimental results on two widely used VS datasets have demonstrated that our proposed methods present clear superiority over existing sparse representation based VS methods and are highly comparable to some deep learning ones requiring supervision information for extra model training.

Research Area(s)

  • Video summarization (VS), Keyframe extraction, Orthogonal Matching Pursuit (OMP), Sparse representation, Block sparse representation

Citation Format(s)

Patch Based Video Summarization With Block Sparse Representation. / Mei, Shaohui; Ma, Mingyang; Wan, Shuai et al.

In: IEEE Transactions on Multimedia, Vol. 23, 9069290, 2021, p. 732-747.

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review