Projects per year
Abstract
Previous work utilized three-dimensional (3-D) convolutional neural networks (CNNs) tomodel the spatial appearance and temporal evolution concurrently for sign language recognition (SLR) and exhibited impressive performance. However, there are still challenges for 3-D CNN-based methods. First, motion information plays a more significant role than spatial content in sign language. Therefore, it is still questionable whether to treat space and time equally and model them jointly by heavy 3-D convolutions in a unified approach. Second, because of the interference from the highly redundant information in sign videos, it is still nontrivial to effectively extract discriminative spatiotemporal features related to sign language. In this study, deep R(2+1)D was adopted for separate spatial and temporal modeling and demonstrated that decomposing 3-D convolution filters into independent spatial and temporal convolutions facilitates the optimization process in SLR. A lightweight spatial–temporal–channel attention module, including two submodules called channel–temporal attention and spatial–temporal attention, was proposed to make the network concentrate on the significant information along spatial, temporal, and channel dimensions by combining squeeze and excitation attention with self-attention. By embedding this module into R(2+1)D, superior or comparable results to the state-of-the-art methods on the CSL-500, Jester, and EgoGesture datasets were obtained, which demonstrated the effectiveness of the proposed method. © 2022 IEEE.
| Original language | English |
|---|---|
| Pages (from-to) | 687-698 |
| Number of pages | 12 |
| Journal | IEEE Transactions on Human-Machine Systems |
| Volume | 52 |
| Issue number | 4 |
| Online published | 3 Feb 2022 |
| DOIs | |
| Publication status | Published - Aug 2022 |
Funding
This work was supported in part by the National Natural Science Foundation of China under Grant 61973187 and Grant 61773239, in part by the Guangdong Province Basic and Applied Basic Research Fund Project under Grant 2019A1515110175, and in part by the Research Grants Council of Hong Kong under Grant 21212720.
Research Keywords
- Convolution
- Feature extraction
- Videos
- Hidden Markov models
- Gesture recognition
- Task analysis
- Spatiotemporal phenomena
- Attention mechanism
- R(2+1)D
- sign language recognition (SLR)
- SCALE GESTURE RECOGNITION
- FRAMEWORK
- FUSION
RGC Funding Information
- RGC-funded
Fingerprint
Dive into the research topics of 'Sign Language Recognition Based on R(2+1)D With Spatial–Temporal–Channel Attention'. Together they form a unique fingerprint.Projects
- 1 Finished
-
ECS: A New Augmented Reality Assisted Image Guidance System for Cardiac Interventions
LIU, J. (Principal Investigator / Project Coordinator)
1/01/21 → 26/02/24
Project: Research
Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver