Rendering-oriented 3D Data Compression: from Unified Representation to Generative Quality Enhancement

The rapid advancement of immersive technologies (e.g., metaverse, digital twins) and the emergence of spatial intelligence are driving an exponential demand for high-fidelity 3D data. However, this growth is severely constrained by the fundamental limitations of existing 3D data compression methods that are necessary for transmission and storage. Specifically, current compression approaches for 3D meshes and point clouds struggle with two interconnected challenges: their unstructured representation leads to inefficient and ineffective spatial redundancy elimination; their separate compression pipelines for geometry and texture, and optimization goals not aligned with the final rendered visual quality, result in inefficient compression and inferior visual fidelity.To overcome these bottlenecks, this project proposes a comprehensive framework consisting of three key components for representation, compression, and quality enhancement:1) We will propose a geometry-texture unified deformable tensor (UniGeT) representation, which adopts deformable distance and color fields to uniformly represent the geometry and texture of a textured 3D mesh as a regular, compact, and sparse volume, capturing fine-grained details and enabling precise surface reconstruction and real-time, differentiable rendering.2) We will build a rendering-oriented neural compression framework that transforms the above UniGeT representation into a compact latent feature and encodes it into bitstreams using a hierarchical context model. Crucially, this compressor is optimized end-to-end by directly minimizing the rendering distortion, ensuring optimal bitrate allocation and control for visual quality.3) We will investigate an effective 3D quality enhancement approach that harnesses the powerful generative capability of a 2D foundation model to recover geometric and textural details of decompressed 3D data suffering from inevitable loss. Particularly, a rendering-centric pipeline will be leveraged to transfer 2D generative priors into targeted 3D model refinement, avoiding the complexities of direct 3D manipulation.By integrating a unified representation with rendering-driven compression and generative enhancement, this project can deliver a breakthrough in compression efficiency and visual fidelity, effectively bridging the gap between the high demands of next-generation immersive applications and the constraints of practical network conditions.Building upon our strong research foundation and encouraging preliminary results, we anticipate that this project will achieve transformative breakthroughs in 3D immersive media representation and compression, significantly advance current technical capabilities, and drive the practical implementation of virtual reality applications. Beyond technical innovation, this project is expected to lay an extensible foundation for future research in spatial data processing and contribute to Hong Kong’s key initiatives in digital entertainment and smart city development.