Multi-stream feature refinement network for human object interaction detection

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review

1 Scopus Citations
View graph of relations


  • Zhanpeng Shao
  • Zhongyan Hu
  • Jianyu Yang
  • Youfu Li

Related Research Unit(s)


Original languageEnglish
Article number103529
Journal / PublicationJournal of Visual Communication and Image Representation
Online published5 May 2022
Publication statusPublished - Jul 2022


Human–Object Interaction (HOI) detection is a crucial problem for comprehensive visual understanding, which aims to detect < human, action, object > triplets within an image. Many existing methods often exploit to integrate the human and object visual features , the spatial layout of human-object pairs, human poses, contextual information, and even object semantic information into a framework to infer the interactions, proving that all these components can contribute to improve the HOI detection. However, most methods simply concatenate these components that are not explicitly embedded in the feature learning for HOI detection. In this paper, we are trying to fuse these components explicitly using a multi-stream feature refinement network. The network extracts the visual features of humans, contexts, and objects, which receives the attentions from human poses, spatial configurations, and semantic prior knowledge of objects to refine these visual features, respectively. In addition, an additional graph neural network is employed here to learn the structural features of human–object pairs. We verify our method on V-COCO and HICO-DET datasets with extensive experiments. The experimental results demonstrate that our method is a simple yet effective for HOI detection, achieving superior performance to those state-of-the-art methods.

Research Area(s)

  • Action recognition, Feature refinement and learning, Human object interaction, Human pose, Multi-stream neural network, Object semantic information