Multiple-input streams attention (MISA) network for skeleton-based construction workers' action recognition using body-segment representation strategies

Research output: Journal Publications and ReviewsRGC 21 - Publication in refereed journalpeer-review

View graph of relations

Author(s)

Detail(s)

Original languageEnglish
Article number105104
Journal / PublicationAutomation in Construction
Volume156
Online published6 Oct 2023
Publication statusPublished - Dec 2023

Abstract

With the rapid growth of deep learning algorithms, graph convolutional networks (GCNs) have become a common choice for skeleton-based human action recognition, boasting impressive performance. However, existing GCN-based models often rely on physical human body connections, which may not suit complex construction tasks involving various body parts and hand movements. To address this concern, the human body is modeled in this paper through topological graphs at varying levels, designed based on body-segment strategies. A multiple-input streams attention (MISA) network is introduced, incorporating GCN and temporal convolutional network (TCN) components to enhance the body-structure topology graph of GCNs with more comprehensive input graphs. Additionally, two-modality motion data and three attention blocks are integrated to capture more discerning features. Finally, experimental results using the Construction Motion Library (CML) dataset demonstrated the superiority of the developed method, reaching approximately 84.94% recognition accuracy. © 2023 Elsevier B.V.

Research Area(s)

  • Attention mechanism, Body-segmentation representation, Construction worker activities, Multi-stream deep learning, Skeleton-based recognition

Citation Format(s)

Multiple-input streams attention (MISA) network for skeleton-based construction workers' action recognition using body-segment representation strategies. / Tian, Yuanyuan; Chen, Jiayu; Kim, Jung In et al.
In: Automation in Construction, Vol. 156, 105104, 12.2023.

Research output: Journal Publications and ReviewsRGC 21 - Publication in refereed journalpeer-review