PointVST : Self-Supervised Pre-Training for 3D Point Clouds Via View-Specific Point-to-Image Translation
Research output: Journal Publications and Reviews › RGC 21 - Publication in refereed journal › peer-review
Author(s)
Related Research Unit(s)
Detail(s)
Original language | English |
---|---|
Pages (from-to) | 6900-6912 |
Number of pages | 13 |
Journal / Publication | IEEE Transactions on Visualization and Computer Graphics |
Volume | 30 |
Issue number | 10 |
Online published | 21 Dec 2023 |
Publication status | Published - Oct 2024 |
Link(s)
Abstract
The past few years have witnessed the great success and prevalence of self-supervised representation learning within the language and 2D vision communities. However, such advancements have not been fully migrated to the field of 3D point cloud learning. Different from existing pre-training paradigms designed for deep point cloud feature extractors that fall into the scope of generative modeling or contrastive learning, this paper proposes a translative pre-training framework, namely PointVST, driven by a novel self-supervised pretext task of cross-modal translation from 3D point clouds to their corresponding diverse forms of 2D rendered images. More specifically, we begin with deducing view-conditioned point- wise embeddings through the insertion of the viewpoint indicator, and then adaptively aggregate a view-specific global codeword, which can be further fed into subsequent 2D convolutional translation heads for image generation. Extensive experimental evaluations on various downstream task scenarios demonstrate that our PointVST shows consistent and prominent performance superiority over current state-of-the-art approaches as well as satisfactory domain transfer capability. Our code will be publicly available at https://github.com/keeganhk/PointVST. © 2023 IEEE.
Research Area(s)
- 3D point clouds, self-supervised learning, multi-view images, pre-training, cross-modal
Bibliographic Note
Research Unit(s) information for this publication is provided by the author(s) concerned.
Citation Format(s)
PointVST: Self-Supervised Pre-Training for 3D Point Clouds Via View-Specific Point-to-Image Translation. / Zhang, Qijian; Hou, Junhui.
In: IEEE Transactions on Visualization and Computer Graphics, Vol. 30, No. 10, 10.2024, p. 6900-6912.
In: IEEE Transactions on Visualization and Computer Graphics, Vol. 30, No. 10, 10.2024, p. 6900-6912.
Research output: Journal Publications and Reviews › RGC 21 - Publication in refereed journal › peer-review