Learning-based Three-dimensional Point Cloud Data Reconstruction and Processing

Description

Three-dimensional point clouds (3DPCs), which consist of a set of 3D coordinates indicating the spatial locations of points to represent the geometric structures of real objects/scenes explicitly, are becoming popular in various emerging applications owing to their flexibility and computational-efficiency. Additionally, the recent advancements of 3D sensing technology make 3D acquisition relatively easy. In spite of its popularity, some fundamental issues still exist, diminishing the use of such data modality. First, it is still costly and time-consuming to acquire high-fidelity dynamic 3DPCs, i.e., a sequence of 3DPCs densely-sampled in both spatial and temporal domains for simultaneously capturing geometric details of objects/scenes and their temporal changes. Instead of relying on better hardware configurations, we will investigate a novel learning-based framework, which is capable of reconstructing such high-fidelity geometry data from a hybrid input, including a static dense 3DPC and a dynamic sparse one. Being aware of the essential difference between 2D image and 3D geometry data as well as the challenges posed by the irregular structure of 3DPCs, we will develop a deep learning framework with novel modules from the perspective of discrete differential geometry to explore and leverage the advantages of the hybrid input in complementary and parallel manners. Second, the associated algorithms for processing such emerging data modality have not been well established. Particularly, as a fundamental theory, sparse representation has been popular and proven to be advantageous in 2D image processing. Unfortunately, the irregular structure of 3DPCs challenges the straightforward extension of existing techniques. In this project, by making full use of the unique permutation-invariant characteristic of such data, we will study novel learning-based permutation-driven approaches for sparsely representing 3DPCs. Besides 3DPCs, the proposed methods are applicable to general high-dimensional, unstructured signals. With our solid backgrounds and promising preliminary verifications achieved, it is highly expected that our investigations will provide effective solutions for rapid and affordable acquisition of high-fidelity dynamic 3DPCs as well as elegant algorithms for processing 3DPCs, which will significantly advance the 3D acquisition field and be beneficial to 3DPC-based applications, included but not limited to effective data transmission for immersive 3D telepresence, accurate object recognition for autonomous driving, visually pleasing perception for virtual/augmented reality, and efficient documentation of culture heritage sites. We believe that beyond the three-year period envisioned for this work, the scientific findings of this project will continuously motivate the research on irregular high-dimensional signal modeling and be applied to real-world applications.