Realistic mouth-synching for speech-driven talking face using articulatory modelling

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review

59 Scopus Citations
View graph of relations

Author(s)

  • Lei Xie
  • Zhi-Qiang Liu

Related Research Unit(s)

Detail(s)

Original languageEnglish
Pages (from-to)500-510
Journal / PublicationIEEE Transactions on Multimedia
Volume9
Issue number3
Publication statusPublished - Apr 2007

Abstract

This paper presents an articulatory modelling approach to convert acoustic speech into realistic mouth animation. We directly model the movements of articulators, such as lips, tongue, and teeth, using a dynamic Bayesian network (DBN)-based audio-visual articulatory model (AVAM). A multiple-stream structure with a shared articulator layer is adopted in the model to synchronously associate the two building blocks of speech, i.e., audio and video. This model not only describes the synchronization between visual articulatory movements and audio speech, but also reflects the linguistic fact that different articulators evolve asynchronously. We also present a Baum-Weich DBN inversion (DBNI) algorithm to generate optimal facial parameters from audio given the trained AVAM under maximum likelihood (ML) criterion. Extensive objective and subjective evaluations on the JEWEL audio-visual dataset demonstrate that compared with phonemic HMM approaches, facial parameters estimated by our approach follow the true parameters more accurately, and the synthesized facial animation sequences are so lively that 38% of them are undistinguishable. © 2007 IEEE.

Research Area(s)

  • Articulatory model, Baum-Welch DBN inversion (DBNI), Dynamic Bayesian networks (DBNs), Facial animation, Mouth-synching, Talking face