Multi-stream feature refinement network for human object interaction detection
Research output: Journal Publications and Reviews (RGC: 21, 22, 62) › 21_Publication in refereed journal › peer-review
Author(s)
Related Research Unit(s)
Detail(s)
Original language | English |
---|---|
Article number | 103529 |
Journal / Publication | Journal of Visual Communication and Image Representation |
Volume | 86 |
Online published | 5 May 2022 |
Publication status | Published - Jul 2022 |
Link(s)
Abstract
Human–Object Interaction (HOI) detection is a crucial problem for comprehensive visual understanding, which aims to detect < human, action, object > triplets within an image. Many existing methods often exploit to integrate the human and object visual features , the spatial layout of human-object pairs, human poses, contextual information, and even object semantic information into a framework to infer the interactions, proving that all these components can contribute to improve the HOI detection. However, most methods simply concatenate these components that are not explicitly embedded in the feature learning for HOI detection. In this paper, we are trying to fuse these components explicitly using a multi-stream feature refinement network. The network extracts the visual features of humans, contexts, and objects, which receives the attentions from human poses, spatial configurations, and semantic prior knowledge of objects to refine these visual features, respectively. In addition, an additional graph neural network is employed here to learn the structural features of human–object pairs. We verify our method on V-COCO and HICO-DET datasets with extensive experiments. The experimental results demonstrate that our method is a simple yet effective for HOI detection, achieving superior performance to those state-of-the-art methods.
Research Area(s)
- Action recognition, Feature refinement and learning, Human object interaction, Human pose, Multi-stream neural network, Object semantic information
Citation Format(s)
Multi-stream feature refinement network for human object interaction detection. / Shao, Zhanpeng; Hu, Zhongyan; Yang, Jianyu et al.
In: Journal of Visual Communication and Image Representation, Vol. 86, 103529, 07.2022.Research output: Journal Publications and Reviews (RGC: 21, 22, 62) › 21_Publication in refereed journal › peer-review