Patch based Video Summarization with Block Sparse Representation

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journal

View graph of relations

Author(s)

  • Shaohui Mei
  • Mingyang Ma
  • Shuai Wan
  • Zhiyong Wang
  • David Dagan Feng

Related Research Unit(s)

Detail(s)

Original languageEnglish
Journal / PublicationIEEE Transactions on Multimedia
Publication statusOnline published - 16 Apr 2020

Abstract

In recent years, sparse representation has been successfully utilized for video summarization (VS). However, most of the sparse representation based VS methods characterize each video frame with global features. As a result, some important local details could be neglected by global features, which may compromise the performance of summarization. In this paper, we propose to partition each video frame into a number of patches and characterize each patch with global features. Instead of concatenating the features of each patch and utilizing conventional sparse representation, we formulate the VS problem with such video frame representation as block sparse representation by considering each video frame as a block containing a number of patches. By taking the reconstruction constraint into account, we devise a simultaneous version of block-based OMP (Orthogonal Matching Pursuit) algorithm, namely SBOMP, to solve the proposed model. The proposed model is further extended to a neighborhood based model which considers temporally adjacent frames as a super block. This is one of the first sparse representation based VS methods taking both spatial and temporal contexts into account with blocks. Experimental results on two widely used VS datasets have demonstrated that our proposed methods present clear superiority over existing sparse representation based VS methods and are highly comparable to some deep learning ones requiring supervision information for extra model training.

Research Area(s)

  • Video summarization (VS), Keyframe extraction, Orthogonal Matching Pursuit (OMP), Sparse representation, Block sparse representation

Citation Format(s)

Patch based Video Summarization with Block Sparse Representation. / Mei, Shaohui; Ma, Mingyang; Wan, Shuai; Hou, Junhui; Wang, Zhiyong; Feng, David Dagan.

In: IEEE Transactions on Multimedia, 16.04.2020.

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journal