Deep Video-Based Performance Cloning

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review

16 Scopus Citations
View graph of relations

Author(s)

  • K. Aberman
  • M. Shi
  • D. Liscbinski
  • B. Chen
  • D. Cohen-Or

Related Research Unit(s)

Detail(s)

Original languageEnglish
Pages (from-to)219-233
Journal / PublicationComputer Graphics Forum
Volume38
Issue number2
Publication statusPublished - May 2019

Abstract

We present a new video-based performance cloning technique. After training a deep generative network using a reference video capturing the appearance and dynamics of a target actor, we are able to generate videos where this actor reenacts other performances. All of the training data and the driving performances are provided as ordinary video segments, without motion capture or depth information. Our generative model is realized as a deep neural network with two branches, both of which train the same space-time conditional generator, using shared weights. One branch, responsible for learning to generate the appearance of the target actor in various poses, uses paired training data, self-generated from the reference video. The second branch uses unpaired data to improve generation of temporally coherent video renditions of unseen pose sequences. Through data augmentation, our network is able to synthesize images of the target actor in poses never captured by the reference video. We demonstrate a variety of promising results, where our method is able to generate temporally coherent videos, for challenging scenarios where the reference and driving videos consist of very different dance performances.

Research Area(s)

  • CCS Concepts, Neural networks, • Computing methodologies → Image-based rendering

Bibliographic Note

Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s)”.

Citation Format(s)

Deep Video-Based Performance Cloning. / Aberman, K.; Shi, M.; Liao, J.; Liscbinski, D.; Chen, B.; Cohen-Or, D.

In: Computer Graphics Forum, Vol. 38, No. 2, 05.2019, p. 219-233.

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review