Supervoxel Convolution for Online 3D Semantic Segmentation

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review

20 Scopus Citations
View graph of relations


  • Shi-Sheng Huang
  • Ze-Yu Ma
  • Tai-Jiang Mu
  • Hongbo Fu
  • Shi-Min Hu

Related Research Unit(s)


Original languageEnglish
Article number34
Journal / PublicationACM Transactions on Graphics
Issue number3
Publication statusPublished - Jul 2021


Online 3D semantic segmentation, which aims to perform real-time 3D scene reconstruction along with semantic segmentation, is an important but challenging topic. A key challenge is to strike a balance between efficiency and segmentation accuracy. There are very fewdeep-learning-based solutions to this problem, since the commonly used deep representations based on volumetric-grids or points do not provide efficient 3D representation and organization structure for online segmentation. Observing that on-surface supervoxels, i.e., clusters of on-surface voxels, provide a compact representation of 3D surfaces and brings efficient connectivity structure via supervoxel clustering, we explore a supervoxel-based deep learning solution for this task. To this end, we contribute a novel convolution operation (SVConv) directly on supervoxels. SVConv can efficiently fuse the multi-view 2D features and 3D features projected on supervoxels during the online 3D reconstruction, and leads to an effective supervoxel-based convolutional neural network, termed as Supervoxel-CNN, enabling 2D-3D joint learning for 3D semantic prediction. With the Supervoxel-CNN, we propose a clustering-then-prediction online 3D semantic segmentation approach. The extensive evaluations on the public 3D indoor scene datasets show that our approach significantly outperforms the existing online semantic segmentation systems in terms of efficiency or accuracy.

Research Area(s)

  • Depth fusion, semantic mapping, supervoxel clustering, supervoxel convolution, deep learning, RECONSTRUCTION, MESHES

Citation Format(s)

Supervoxel Convolution for Online 3D Semantic Segmentation. / Huang, Shi-Sheng; Ma, Ze-Yu; Mu, Tai-Jiang et al.
In: ACM Transactions on Graphics, Vol. 40, No. 3, 34, 07.2021.

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review