DeepPortraitDrawing : Generating human body images from freehand sketches

Research output: Journal Publications and ReviewsRGC 21 - Publication in refereed journalpeer-review

1 Scopus Citations
View graph of relations

Author(s)

  • Xian Wu
  • Chen Wang
  • Hongbo Fu
  • Ariel Shamir
  • Song-Hai Zhang

Related Research Unit(s)

Detail(s)

Original languageEnglish
Pages (from-to)73-81
Journal / PublicationComputers and Graphics (Pergamon)
Volume116
Online published9 Aug 2023
Publication statusPublished - Nov 2023

Abstract

Various methods for generating realistic images of objects and human faces from freehand sketches have been explored. However, generating realistic human body images from sketches is still a challenging problem. It is, first because of the sensitivity to human shapes, second because of the complexity of human images caused by body shape and pose changes, and third because of the domain gap between realistic images and freehand sketches. In this work, we present DeepPortraitDrawing, a deep generative framework for converting roughly drawn sketches to realistic human body images. To encode complicated body shapes under various poses, we take a local-to-global approach. Locally, we employ semantic part auto-encoders to construct part-level shape spaces, which are useful for refining the geometry of an input pre-segmented hand-drawn sketch. Globally, we employ a cascaded spatial transformer network to refine the structure of body parts by adjusting their spatial locations and relative proportions. Finally, we use a style-based generator as the global synthesis network for the sketch-to-image translation task which is modulated by segmentation maps for semantic preservation. Extensive experiments have shown that given roughly sketched human portraits, our method produces more realistic images than the state-of-the-art sketch-to-image synthesis techniques. © 2023 Elsevier Ltd

Research Area(s)

  • Generative adversarial networks, Image-to-image generation, StyleGAN

Bibliographic Note

Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s).

Citation Format(s)

DeepPortraitDrawing: Generating human body images from freehand sketches. / Wu, Xian; Wang, Chen; Fu, Hongbo et al.
In: Computers and Graphics (Pergamon), Vol. 116, 11.2023, p. 73-81.

Research output: Journal Publications and ReviewsRGC 21 - Publication in refereed journalpeer-review