Towards Practical 3D Human Reconstruction and Rendering: Occlusion Robustness, Multi-Person Modeling, and Mobile Efficiency

Abstract

Recovering and rendering 3D humans are important problems for digital human animation, augmented reality, gaming, motion analysis, and human-computer interaction. Parametric human models and deep neural networks have made substantial progress in reconstructing a 3D human mesh from a single RGB image, but practical use remains difficult because natural images often provide incomplete and ambiguous evidence. A target person may be partially hidden by self-occlusion, external objects, or other people. Human-human occlusion is a particularly challenging case: the occluder is another articulated body with plausible limbs and joints, so the model must infer missing regions while also preserving instance identity and body-part association. In addition, many existing systems depend on external detectors and cropped inputs, making their performance sensitive to detection quality. Practical applications further require more than body pose and shape; hand gestures, facial expressions, efficient inference, and real-time display on consumer devices are also important. This thesis addresses these connected requirements through four studies on occlusion-aware reconstruction, efficient structure-aware completion, detection-free multi-person expressive recovery, and real-time mobile rendering.

The first work introduces Dense Inpainting Human Mesh Recovery (DIMR), a dense UV completion framework for human mesh recovery under severe occlusion. DIMR addresses two difficulties caused by occlusion: separating target-human evidence from occluders and inferring features for invisible body regions. It projects visible image features into a canonical UV representation using dense IUV correspondence, and then completes missing body-part features with an attention-based module. A feature inpainting training strategy provides direct supervision for this completion process. The results show that dense correspondence can be used not only for alignment, but also as a structured space for occlusion-aware reconstruction.

The second work, Occlusion-aware Structure Inpainting for Robust Human Mesh Recovery (OSIR), continues the occlusion study with a stronger emphasis on efficiency. OSIR combines query selection, dense IUV estimation, UV-space feature warping, symmetry-aware inpainting, and mask-guided refinement. By selecting human-centric image features before transformer decoding and completing missing regions in canonical UV space, OSIR reduces unnecessary computation while keeping the benefit of dense surface reasoning. This improves the accuracy-efficiency trade-off for human mesh recovery in occluded scenes.

The third work studies multi-person whole-body human pose and shape estimation with AiOS, a detection-free all-in-one-stage framework for SMPL-X recovery. Instead of relying on external human detectors, body-part detectors, or cropped inputs, AiOS estimates all people directly from the full image. It jointly recovers body pose, hand gestures, facial expressions, and body shape for multiple people. By formulating expressive mesh recovery as a progressive set prediction problem with human-related and joint-related tokens, AiOS preserves scene context, models relationships among people, and improves whole-body consistency across body, hands, and face. Inter-human occlusion is treated as an important crowded-scene case rather than the only setting addressed by the method.

The fourth work moves from reconstruction to deployment by studying real-time lighting for consumer-grade mobile graphics hardware. Reconstructed or animated humans are useful only when they can be displayed interactively on practical devices. The proposed rendering framework combines concentric spherical representation, seamless dual paraboloid map mipmap, array-texture storage, and environment-map approximation with multiple omnidirectional light sources. It produces visually plausible soft shadows and reflection effects under mobile memory and computation constraints, addressing the efficient presentation stage of practical 3D human applications.

Overall, this thesis investigates practical 3D human reconstruction and rendering rather than treating mesh recovery only as a benchmark regression task. DIMR and OSIR make the reconstruction stage more robust to occlusion, AiOS removes the dependence on a detector-and-crop pipeline for multi-person SMPL-X recovery, and the rendering work addresses how animated 3D content can be displayed on consumer devices. The four studies cover surface-structured reconstruction, scene- and instance-level reasoning, expressive modeling, and real-time presentation.

Date of Award	30 Jul 2026
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Chi Sing Andrew LEUNG (Supervisor) & Chak Chung Ray CHEUNG (Co-supervisor)