Skip to main navigation Skip to search Skip to main content

Sign Language Recognition Based on R(2+1)D With Spatial–Temporal–Channel Attention

  • Xiangzu Han
  • , Fei Lu*
  • , Jianqin Yin
  • , Guohui Tian
  • , Jun Liu*
  • *Corresponding author for this work

Research output: Journal Publications and Reviews › RGC 21 - Publication in refereed journal › peer-review

Abstract

Previous work utilized three-dimensional (3-D) convolutional neural networks (CNNs) tomodel the spatial appearance and temporal evolution concurrently for sign language recognition (SLR) and exhibited impressive performance. However, there are still challenges for 3-D CNN-based methods. First, motion information plays a more significant role than spatial content in sign language. Therefore, it is still questionable whether to treat space and time equally and model them jointly by heavy 3-D convolutions in a unified approach. Second, because of the interference from the highly redundant information in sign videos, it is still nontrivial to effectively extract discriminative spatiotemporal features related to sign language. In this study, deep R(2+1)D was adopted for separate spatial and temporal modeling and demonstrated that decomposing 3-D convolution filters into independent spatial and temporal convolutions facilitates the optimization process in SLR. A lightweight spatial–temporal–channel attention module, including two submodules called channel–temporal attention and spatial–temporal attention, was proposed to make the network concentrate on the significant information along spatial, temporal, and channel dimensions by combining squeeze and excitation attention with self-attention. By embedding this module into R(2+1)D, superior or comparable results to the state-of-the-art methods on the CSL-500, Jester, and EgoGesture datasets were obtained, which demonstrated the effectiveness of the proposed method. © 2022 IEEE.
Original languageEnglish
Pages (from-to)687-698
Number of pages12
JournalIEEE Transactions on Human-Machine Systems
Volume52
Issue number4
Online published3 Feb 2022
DOIs
Publication statusPublished - Aug 2022

Funding

This work was supported in part by the National Natural Science Foundation of China under Grant 61973187 and Grant 61773239, in part by the Guangdong Province Basic and Applied Basic Research Fund Project under Grant 2019A1515110175, and in part by the Research Grants Council of Hong Kong under Grant 21212720.

Research Keywords

  • Convolution
  • Feature extraction
  • Videos
  • Hidden Markov models
  • Gesture recognition
  • Task analysis
  • Spatiotemporal phenomena
  • Attention mechanism
  • R(2+1)D
  • sign language recognition (SLR)
  • SCALE GESTURE RECOGNITION
  • FRAMEWORK
  • FUSION

RGC Funding Information

  • RGC-funded

Fingerprint

Dive into the research topics of 'Sign Language Recognition Based on R(2+1)D With Spatial–Temporal–Channel Attention'. Together they form a unique fingerprint.

Cite this