Cross-Modal and Cross-Domain Knowledge Transfer for Label-Free 3D Segmentation

Jingyu Zhang; Huitong Yang; Dai-Jie Wu; Jacky Keung; Xuesong Li; Xinge Zhu; Yuexin Ma

doi:10.1007/978-981-99-8435-0_37

Cross-Modal and Cross-Domain Knowledge Transfer for Label-Free 3D Segmentation

Jingyu Zhang
, Huitong Yang
, Dai-Jie Wu
, Jacky Keung
, Xuesong Li
, Xinge Zhu^*
, Yuexin Ma^*

^*Corresponding author for this work

Department of Computer Science

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

2 Citations (Scopus)

Abstract

Current state-of-the-art point cloud-based perception methods usually rely on large-scale labeled data, which requires expensive manual annotations. A natural option is to explore the unsupervised methodology for 3D perception tasks. However, such methods of tenface substantial performance-drop difficulties. Fortunately, we found that there exist amounts of image-based datasets and an alternative can be proposed, i.e., transferring the knowledge in the 2D images to 3D point clouds. Specifically, we propose a novel approach for the challenging cross-modal and cross-domain adaptation task by fully exploring the relationship between images and point clouds and designing effective feature alignment strategies. Without any 3D labels, our method achieves state-of-the-art performance for 3D point cloud semantic segmentation on SemanticKITTI by using the knowledge of KITTI360 and GTA5,compared to existing unsupervised and weakly-supervised baselines.

© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024Q. Liu et al. (Eds.): PRCV 2023, LNCS 14427, pp. 465–477, 2024.

Original language	English
Title of host publication	Pattern Recognition and Computer Vision
Subtitle of host publication	6th Chinese Conference, PRCV 2023, Proceedings, Part III
Editors	Qingshan Liu, Hanzi Wang, Zhanyu Ma, Weishi Zheng, Hongbin Zha, Xilin Chen, Liang Wang, Rongrong Ji
Publisher	Springer Singapore
Pages	465-477
Edition	1
ISBN (Electronic)	978-981-99-8435-0
ISBN (Print)	978-981-99-8434-3
DOIs	https://doi.org/10.1007/978-981-99-8435-0_37
Publication status	Published - Oct 2023
Event	6th Chinese Conference on Pattern Recognition and Computer Vision (PRCV 2023) - Xiamen, China Duration: 13 Oct 2023 → 15 Oct 2023 http://prcv2023.xmu.edu.cn

Publication series

Name	Lecture Notes in Computer Science
Volume	14427
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	6th Chinese Conference on Pattern Recognition and Computer Vision (PRCV 2023)
Place	China
City	Xiamen
Period	13/10/23 → 15/10/23
Internet address	http://prcv2023.xmu.edu.cn

Research Keywords

Point Cloud Semantic Segmentation
Unsupervised Domain Adaptation
Cross-modal Transfer Learning

Access Output

10.1007/978-981-99-8435-0_37

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Zhang, J., Yang, H., Wu, D.-J., Keung, J., Li, X., Zhu, X., & Ma, Y. (2023). Cross-Modal and Cross-Domain Knowledge Transfer for Label-Free 3D Segmentation. In Q. Liu, H. Wang, Z. Ma, W. Zheng, H. Zha, X. Chen, L. Wang, & R. Ji (Eds.), Pattern Recognition and Computer Vision: 6th Chinese Conference, PRCV 2023, Proceedings, Part III (1 ed., pp. 465-477). (Lecture Notes in Computer Science; Vol. 14427). Springer Singapore. https://doi.org/10.1007/978-981-99-8435-0_37

Zhang, Jingyu ; Yang, Huitong ; Wu, Dai-Jie et al. / Cross-Modal and Cross-Domain Knowledge Transfer for Label-Free 3D Segmentation. Pattern Recognition and Computer Vision: 6th Chinese Conference, PRCV 2023, Proceedings, Part III. editor / Qingshan Liu ; Hanzi Wang ; Zhanyu Ma ; Weishi Zheng ; Hongbin Zha ; Xilin Chen ; Liang Wang ; Rongrong Ji. 1. ed. Springer Singapore, 2023. pp. 465-477 (Lecture Notes in Computer Science).

@inproceedings{20184a40908541d7bc91fdafff03919f,

title = "Cross-Modal and Cross-Domain Knowledge Transfer for Label-Free 3D Segmentation",

abstract = "Current state-of-the-art point cloud-based perception methods usually rely on large-scale labeled data, which requires expensive manual annotations. A natural option is to explore the unsupervised methodology for 3D perception tasks. However, such methods of tenface substantial performance-drop difficulties. Fortunately, we found that there exist amounts of image-based datasets and an alternative can be proposed, i.e., transferring the knowledge in the 2D images to 3D point clouds. Specifically, we propose a novel approach for the challenging cross-modal and cross-domain adaptation task by fully exploring the relationship between images and point clouds and designing effective feature alignment strategies. Without any 3D labels, our method achieves state-of-the-art performance for 3D point cloud semantic segmentation on SemanticKITTI by using the knowledge of KITTI360 and GTA5,compared to existing unsupervised and weakly-supervised baselines. {\textcopyright} The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024Q. Liu et al. (Eds.): PRCV 2023, LNCS 14427, pp. 465–477, 2024.",

keywords = "Point Cloud Semantic Segmentation, Unsupervised Domain Adaptation, Cross-modal Transfer Learning",

author = "Jingyu Zhang and Huitong Yang and Dai-Jie Wu and Jacky Keung and Xuesong Li and Xinge Zhu and Yuexin Ma",

year = "2023",

month = oct,

doi = "10.1007/978-981-99-8435-0\_37",

language = "English",

isbn = "978-981-99-8434-3",

series = "Lecture Notes in Computer Science",

publisher = "Springer Singapore",

pages = "465--477",

editor = "Qingshan Liu and Hanzi Wang and Zhanyu Ma and Weishi Zheng and Hongbin Zha and Xilin Chen and Liang Wang and Rongrong Ji",

booktitle = "Pattern Recognition and Computer Vision",

address = "Singapore",

edition = "1",

note = "6th Chinese Conference on Pattern Recognition and Computer Vision (PRCV 2023) ; Conference date: 13-10-2023 Through 15-10-2023",

url = "http://prcv2023.xmu.edu.cn",

}

Zhang, J, Yang, H, Wu, D-J, Keung, J, Li, X, Zhu, X & Ma, Y 2023, Cross-Modal and Cross-Domain Knowledge Transfer for Label-Free 3D Segmentation. in Q Liu, H Wang, Z Ma, W Zheng, H Zha, X Chen, L Wang & R Ji (eds), Pattern Recognition and Computer Vision: 6th Chinese Conference, PRCV 2023, Proceedings, Part III. 1 edn, Lecture Notes in Computer Science, vol. 14427, Springer Singapore, pp. 465-477, 6th Chinese Conference on Pattern Recognition and Computer Vision (PRCV 2023), Xiamen, China, 13/10/23. https://doi.org/10.1007/978-981-99-8435-0_37

Cross-Modal and Cross-Domain Knowledge Transfer for Label-Free 3D Segmentation. / Zhang, Jingyu; Yang, Huitong; Wu, Dai-Jie et al.
Pattern Recognition and Computer Vision: 6th Chinese Conference, PRCV 2023, Proceedings, Part III. ed. / Qingshan Liu; Hanzi Wang; Zhanyu Ma; Weishi Zheng; Hongbin Zha; Xilin Chen; Liang Wang; Rongrong Ji. 1. ed. Springer Singapore, 2023. p. 465-477 (Lecture Notes in Computer Science; Vol. 14427).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Cross-Modal and Cross-Domain Knowledge Transfer for Label-Free 3D Segmentation

AU - Zhang, Jingyu

AU - Yang, Huitong

AU - Wu, Dai-Jie

AU - Keung, Jacky

AU - Li, Xuesong

AU - Zhu, Xinge

AU - Ma, Yuexin

PY - 2023/10

Y1 - 2023/10

N2 - Current state-of-the-art point cloud-based perception methods usually rely on large-scale labeled data, which requires expensive manual annotations. A natural option is to explore the unsupervised methodology for 3D perception tasks. However, such methods of tenface substantial performance-drop difficulties. Fortunately, we found that there exist amounts of image-based datasets and an alternative can be proposed, i.e., transferring the knowledge in the 2D images to 3D point clouds. Specifically, we propose a novel approach for the challenging cross-modal and cross-domain adaptation task by fully exploring the relationship between images and point clouds and designing effective feature alignment strategies. Without any 3D labels, our method achieves state-of-the-art performance for 3D point cloud semantic segmentation on SemanticKITTI by using the knowledge of KITTI360 and GTA5,compared to existing unsupervised and weakly-supervised baselines. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024Q. Liu et al. (Eds.): PRCV 2023, LNCS 14427, pp. 465–477, 2024.

AB - Current state-of-the-art point cloud-based perception methods usually rely on large-scale labeled data, which requires expensive manual annotations. A natural option is to explore the unsupervised methodology for 3D perception tasks. However, such methods of tenface substantial performance-drop difficulties. Fortunately, we found that there exist amounts of image-based datasets and an alternative can be proposed, i.e., transferring the knowledge in the 2D images to 3D point clouds. Specifically, we propose a novel approach for the challenging cross-modal and cross-domain adaptation task by fully exploring the relationship between images and point clouds and designing effective feature alignment strategies. Without any 3D labels, our method achieves state-of-the-art performance for 3D point cloud semantic segmentation on SemanticKITTI by using the knowledge of KITTI360 and GTA5,compared to existing unsupervised and weakly-supervised baselines. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024Q. Liu et al. (Eds.): PRCV 2023, LNCS 14427, pp. 465–477, 2024.

KW - Point Cloud Semantic Segmentation

KW - Unsupervised Domain Adaptation

KW - Cross-modal Transfer Learning

UR - https://www.scopus.com/pages/publications/85180807628

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85180807628&origin=recordpage

U2 - 10.1007/978-981-99-8435-0_37

DO - 10.1007/978-981-99-8435-0_37

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 978-981-99-8434-3

T3 - Lecture Notes in Computer Science

SP - 465

EP - 477

BT - Pattern Recognition and Computer Vision

A2 - Liu, Qingshan

A2 - Wang, Hanzi

A2 - Ma, Zhanyu

A2 - Zheng, Weishi

A2 - Zha, Hongbin

A2 - Chen, Xilin

A2 - Wang, Liang

A2 - Ji, Rongrong

PB - Springer Singapore

T2 - 6th Chinese Conference on Pattern Recognition and Computer Vision (PRCV 2023)

Y2 - 13 October 2023 through 15 October 2023

ER -

Zhang J, Yang H, Wu DJ, Keung J, Li X, Zhu X et al. Cross-Modal and Cross-Domain Knowledge Transfer for Label-Free 3D Segmentation. In Liu Q, Wang H, Ma Z, Zheng W, Zha H, Chen X, Wang L, Ji R, editors, Pattern Recognition and Computer Vision: 6th Chinese Conference, PRCV 2023, Proceedings, Part III. 1 ed. Springer Singapore. 2023. p. 465-477. (Lecture Notes in Computer Science). doi: 10.1007/978-981-99-8435-0_37

Cross-Modal and Cross-Domain Knowledge Transfer for Label-Free 3D Segmentation

Abstract

Publication series

Conference

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this