SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling

Zhongjin Luo; Dong Du; Heming Zhu; Yizhou Yu; Hongbo Fu; Xiaoguang Han

doi:10.1109/TVCG.2023.3291703

SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling

Zhongjin Luo, Dong Du, Heming Zhu, Yizhou Yu, Hongbo Fu, Xiaoguang Han^*

^*Corresponding author for this work

School of Creative Media

Research output: Journal Publications and Reviews › RGC 21 - Publication in refereed journal › peer-review

7 Citations (Scopus)

938 Downloads (CityUHK Scholars)

Abstract

Modeling 3D avatars benefits various application scenarios such as AR/VR, gaming, and filming. Character faces contribute significant diversity and vividity as a vital component of avatars. However, building 3D character face models usually requires a heavy workload with commercial tools, even for experienced artists. Various existing sketch-based tools fail to support amateurs in modeling diverse facial shapes and rich geometric details. In this article, we present SketchMetaFace - a sketching system targeting amateur users to model high-fidelity 3D faces in minutes. We carefully design both the user interface and the underlying algorithm. First, curvature-aware strokes are adopted to better support the controllability of carving facial details. Second, considering the key problem of mapping a 2D sketch map to a 3D model, we develop a novel learning-based method termed “Implicit and Depth Guided Mesh Modeling” (IDGMM). It fuses the advantages of mesh, implicit, and depth representations to achieve high-quality results with high efficiency. In addition, to further support usability, we present a coarse-to-fine 2D sketching interface design and a data-driven stroke suggestion tool. User studies demonstrate the superiority of our system over existing modeling tools in terms of the ease to use and visual quality of results. Experimental analyses also show that IDGMM reaches a better trade-off between accuracy and efficiency. © 2023 IEEE.

Original language	English
Pages (from-to)	5260-5275
Number of pages	16
Journal	IEEE Transactions on Visualization and Computer Graphics
Volume	30
Issue number	8
Online published	19 Jul 2023
DOIs	https://doi.org/10.1109/TVCG.2023.3291703
Publication status	Published - Aug 2024

Funding

This work was supported in part by NSFC under Grant 62172348, in part by the Basic Research Project under Grant HZQB-KCZYZ-2021067 through project Hetao Shenzhen-HK S&T Cooperation Zone, in part by the National Key R&D Program of China under Grant 2018YFB1800800, in part by the Shenzhen Outstanding Talents Training Fund under Grant 202002, in part by the Guangdong Research Projects under Grants 2017ZT07X152 and 2019CX01X104, in part by the Guangdong Provincial Key Laboratory of Future Networks of Intelligence under Grant 2022B1212010001, in part by the Shenzhen Key Laboratory of Big Data and Artificial Intelligence under Grant ZDSYS201707251409055, in part by the Key Area R&D Program of Guangdong Province under Grant 2018B030338001, in part by Outstanding Yound Fund of Guangdong Province under Grant 2023B1515020055, in part by Shenzhen General Project under Grant JCYJ20220530143604010, in part by Hong Kong Research Grants Council under General Research Funds under Grant HKU17206218, in part by Research Grants Council of the Hong Kong Special Administrative Region, China under Grant CityU 11212119, and in part by the Centre for Applied Computing and Interactive Media (ACIM) of School of Creative Media, CityU.

Research Keywords

Face Modeling
Neural Network
Sketch-based 3D Modeling

Publisher's Copyright Statement

COPYRIGHT TERMS OF DEPOSITED POSTPRINT FILE: © 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Luo, Z., Du, D., Zhu, H., Yu, Y., Fu, H., & Han, X. (2023). SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling. IEEE Transactions on Visualization and Computer Graphics. Advance online publication. https://doi.org/10.1109/TVCG.2023.3291703

RGC Funding Information

RGC-funded

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access Output

10.1109/TVCG.2023.3291703

163533915.pdfPost-print, 17.5 MB

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

@article{f8383e9c913c4b16a0af8121435f7aa8,

title = "SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling",

abstract = "Modeling 3D avatars benefits various application scenarios such as AR/VR, gaming, and filming. Character faces contribute significant diversity and vividity as a vital component of avatars. However, building 3D character face models usually requires a heavy workload with commercial tools, even for experienced artists. Various existing sketch-based tools fail to support amateurs in modeling diverse facial shapes and rich geometric details. In this article, we present SketchMetaFace - a sketching system targeting amateur users to model high-fidelity 3D faces in minutes. We carefully design both the user interface and the underlying algorithm. First, curvature-aware strokes are adopted to better support the controllability of carving facial details. Second, considering the key problem of mapping a 2D sketch map to a 3D model, we develop a novel learning-based method termed “Implicit and Depth Guided Mesh Modeling” (IDGMM). It fuses the advantages of mesh, implicit, and depth representations to achieve high-quality results with high efficiency. In addition, to further support usability, we present a coarse-to-fine 2D sketching interface design and a data-driven stroke suggestion tool. User studies demonstrate the superiority of our system over existing modeling tools in terms of the ease to use and visual quality of results. Experimental analyses also show that IDGMM reaches a better trade-off between accuracy and efficiency. {\textcopyright} 2023 IEEE.",

keywords = "Face Modeling, Neural Network, Sketch-based 3D Modeling",

author = "Zhongjin Luo and Dong Du and Heming Zhu and Yizhou Yu and Hongbo Fu and Xiaoguang Han",

year = "2024",

month = aug,

doi = "10.1109/TVCG.2023.3291703",

language = "English",

volume = "30",

pages = "5260--5275",

journal = "IEEE Transactions on Visualization and Computer Graphics",

issn = "1077-2626",

publisher = "IEEE Computer Society",

number = "8",

}

TY - JOUR

T1 - SketchMetaFace

T2 - A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling

AU - Luo, Zhongjin

AU - Du, Dong

AU - Zhu, Heming

AU - Yu, Yizhou

AU - Fu, Hongbo

AU - Han, Xiaoguang

PY - 2024/8

Y1 - 2024/8

N2 - Modeling 3D avatars benefits various application scenarios such as AR/VR, gaming, and filming. Character faces contribute significant diversity and vividity as a vital component of avatars. However, building 3D character face models usually requires a heavy workload with commercial tools, even for experienced artists. Various existing sketch-based tools fail to support amateurs in modeling diverse facial shapes and rich geometric details. In this article, we present SketchMetaFace - a sketching system targeting amateur users to model high-fidelity 3D faces in minutes. We carefully design both the user interface and the underlying algorithm. First, curvature-aware strokes are adopted to better support the controllability of carving facial details. Second, considering the key problem of mapping a 2D sketch map to a 3D model, we develop a novel learning-based method termed “Implicit and Depth Guided Mesh Modeling” (IDGMM). It fuses the advantages of mesh, implicit, and depth representations to achieve high-quality results with high efficiency. In addition, to further support usability, we present a coarse-to-fine 2D sketching interface design and a data-driven stroke suggestion tool. User studies demonstrate the superiority of our system over existing modeling tools in terms of the ease to use and visual quality of results. Experimental analyses also show that IDGMM reaches a better trade-off between accuracy and efficiency. © 2023 IEEE.

AB - Modeling 3D avatars benefits various application scenarios such as AR/VR, gaming, and filming. Character faces contribute significant diversity and vividity as a vital component of avatars. However, building 3D character face models usually requires a heavy workload with commercial tools, even for experienced artists. Various existing sketch-based tools fail to support amateurs in modeling diverse facial shapes and rich geometric details. In this article, we present SketchMetaFace - a sketching system targeting amateur users to model high-fidelity 3D faces in minutes. We carefully design both the user interface and the underlying algorithm. First, curvature-aware strokes are adopted to better support the controllability of carving facial details. Second, considering the key problem of mapping a 2D sketch map to a 3D model, we develop a novel learning-based method termed “Implicit and Depth Guided Mesh Modeling” (IDGMM). It fuses the advantages of mesh, implicit, and depth representations to achieve high-quality results with high efficiency. In addition, to further support usability, we present a coarse-to-fine 2D sketching interface design and a data-driven stroke suggestion tool. User studies demonstrate the superiority of our system over existing modeling tools in terms of the ease to use and visual quality of results. Experimental analyses also show that IDGMM reaches a better trade-off between accuracy and efficiency. © 2023 IEEE.

KW - Face Modeling

KW - Neural Network

KW - Sketch-based 3D Modeling

UR - http://www.scopus.com/inward/record.url?scp=85165264177&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85165264177&origin=recordpage

U2 - 10.1109/TVCG.2023.3291703

DO - 10.1109/TVCG.2023.3291703

M3 - RGC 21 - Publication in refereed journal

SN - 1077-2626

VL - 30

SP - 5260

EP - 5275

JO - IEEE Transactions on Visualization and Computer Graphics

JF - IEEE Transactions on Visualization and Computer Graphics

IS - 8

ER -

SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling

Abstract

Funding

Research Keywords

Publisher's Copyright Statement

RGC Funding Information

UN SDGs

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

GRF: Towards Bridging the Gap Between Freehand Sketches and 3D Models

Cite this