Speech and Expression Driven Animation of a Video-Realistic Appearance Based Hierarchical Facial Model

We describe a new facial animation system based on a hierarchy of morphable sub-facial appearance models. The innovation in our approach is that through the hierarchical model, parametric control is available for the animation of multiple sub-facial areas. We animate these areas automatically both from speech to produce lip-synching, and natural pauses and hesitations and using specific temporal variation of appearance parameters to control sub-facial behaviours. Image-based and morphable models are capable of producing highly-realistic facial animations, but currently only provide parametric control of limited facial areas [3]. We are able to produce realism comparable to that of real video, while also providing parameters for animation usually only associated with 3D parametric models. Speech driven animation is facilitated using a Hidden Markov Coarticulation Model (HMCM). The model learns visual speech relationships from a training corpus of 2D video, and creates new visual parameters given new speech – which may not have appeared in the original corpus. For control over non-speech related animation and also for greater control over certain facial expressions in general, parameters responsible for controlling the sub facial movement of the mouth, eyes and eyebrows are extracted from the hierarchy and specific poses identified. These are interpolated to create sub-facial animations for behaviours such as smiling, blinking and winking, and expressions akin to anger, fear, shock and surprise.