Motif-GCNs with Local and Non-Local Temporal Blocks for Skeleton-Based Action Recognition

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review

3 Scopus Citations
View graph of relations


  • Yu-Hui Wen
  • Lin Gao
  • Fang-Lue Zhang
  • Shihong Xia
  • Yong-Jin Liu

Related Research Unit(s)


Original languageEnglish
Pages (from-to)2009-2023
Journal / PublicationIEEE Transactions on Pattern Analysis and Machine Intelligence
Issue number2
Online published26 Apr 2022
Publication statusPublished - Feb 2023


Recent works have achieved remarkable performance for action recognition with human skeletal data by utilizing graph convolutional models. Existing models mainly focus on developing graph convolutions to encode structural properties of the skeletal graph. Some recent works further take sample-dependent relationships among joints into consideration. However, the complex relationships are difficult to learn. In this paper, we propose a motif-based graph convolution method, which makes use of sample-dependent latent relations among non-physically connected joints to impose a high-order locality and assigns different semantic roles to physical neighbors of a joint to encode hierarchical structures. Furthermore, we propose a sparsity-promoting loss function to learn a sparse motif adjacency matrix for latent dependencies in non-physical connections. For extracting effective temporal information, we propose an efficient local temporal block. It adopts partial dense connections to reuse temporal features in local time windows, and enrich a variety of information flow by gradient combination. In addition, we introduce a non-local temporal block to capture global dependencies among frames. Comprehensive experiments on four large-scale datasets show that our model outperforms the state-of-the-art methods. Our code is publicly available at

Research Area(s)

  • Action Recognition, Convolutional codes, Feature extraction, Graph Convolutional Neural Networks, Joints, Non-Local Block, Skeleton, Skeleton Sequence, Sparse matrices, Spatio-Temporal Attention, Topology, Training

Citation Format(s)