Random-Sets-based Higher-Order Hidden Markov Model (RS-HO-HMM) for Stylistic Human Motion Modeling

Description

In this project, we will develop a system for natural stylistic 2D/3D human motion modeling from video sequences and motion capture data. Despite significant research and development efforts during the last few decades, effective modeling and efficient creating stylistic character motion remains one of the main challenges in computer animation and computer vision due to the variability and flexibility of human behaviors. Human motion essentially contains spatial and temporal random variations which can be modeled and synthesized as motion data with a unique style. Recent data-driven techniques facilitate modeling and synthesizing of high-fidelity human motions. However, on the one hand, motion capture data is usually very time-consuming and expensive to acquire in many disciplines. A large repository of recorded motions is difficult to modify and restricted in terms of the available motion styles. One the other hand, many video sequence data are available online due to the rapid popularization of digital video equipments. It is highly desirable to learn generalized motion style models from video sequences and available motion capture data, then providing an effective and principled way to edit and transfer the motion styles.The higher-order Markov random fields (HO-MRFs) and the random sets theory show great promises for modeling and synthesizing highly stylistic motions with a moderate set of data. Specifically, a human motion (e.g. walk) with a variety of styles (e.g., normal, catwalk, strut, crouch, limp) can be naturally interpreted by a random set of higher-order MRFs. The random sets theory is appropriate for modeling natural phenomena in terms of sets rather than precise points. In addition, hidden Markov model (HMM) with higherorder cliques is a powerful tool for modeling sequential changing behaviors and the rich spatial structural dependencies. Hence, in this project, we propose a new branch of hidden Markov models, namely random-sets-based higher-order hidden Markov models (RS-HO-HMMs), for learning and synthesizing stylistic human motions from a modest amount of motion capture data and video sequences.We anticipate that, with the combination of higher-order MRFs, random sets theory and the hidden Markov models, we will be able to achieve a superior performance for modeling, synthesizing, editing and transferring stylistic human motions. Through this study we expect to achieve the following goals:Establish theoretical foundation, benchmark datasets and key algorithms (e.g., learning, decoding and evaluation) for the proposed random-sets-based higherorder hidden Markov models (RS-HO-HMMs);Design random-set-valued higher-order hidden states for learning stylistic human motions from a modest amount of motion capture data and video sequences, and random-set-valued observations for modeling imprecise/imperfect motion capture data and video sequences;Effective coarsening schemes for RS-HO-HMMs, e.g., the maximum entropy based scheme, generalized Gibbs energy minimization, and the CAR model;Stylistic motion synthesis and user-friendly motion editing/transferring techniques based on the learnt RS-HO-HMMs;A human motion synthesis and editing system that can generate and edit/transfer a range of stylistic human body and facial animation.In addition the techniques developed in this project can be readily applied to other broad areas where statistical modeling of structured, dynamical and sequential data is required, for instance, speech, image and video data analysis, face and gesture recognition for games and various interactive applications.