In the paper, we present a framework of speech driven face animation system with expressions. It systematically addresses audio-visual data acquisition, expressive trajectory analysis and audio-visual mapping. Based on this framework, we learn the correlation between neutral facial deformation and expressive facial deformation with Gaussian Mixture Model (GMM). A hierarchical structure is propo...

In this paper, we present a new text-driven avatar system, which consists of three major components, a text-to-speech (TTS) unit, a speech driven facial animation (SDFA) unit and a text-to-sign language (TTSL) unit. A new visual prosody time control model and an integrated learning framework are proposed to realize synchronization among speech synthesis, face animation and gesture animation, wh...

Digital Interactive Storytelling (DIS) is a relatively novel area of computer entertainment which aims at investigating interactive applications capable of generating consistent, emergent and rich stories. To provide new solutions for DIS, we have proposed a novel multi-agent DIS framework, DIEGESIS, which includes agents' coordination and new planning and replanning solutions for DIS. A natura...

Speech is powerful information technology and the basis of human interaction. By emitting streams of buzzing, popping, and hissing noises from our mouths, we transmit thoughts, intentions, and knowledge of the world from one mind to another. We’re accustomed to thinking of speech as an acoustic, auditory phenomenon. However, speech is also visible. Although the primary function of speech is to ...

Facial animation has been a topic of intensive research for more than three decades. Still, designing realistic facial animations remains to be a challenging task. Several models and tools have been developed so far to automate the design of faces and facial animations synchronized with speech, emotions, and gestures. In this article, we take a brief overview of the existing parameterized facia...

The goal of this project is to create a muscle-based facial model which can be used to generate realistic facial animation. Maya software was used to set up the facial muscles, and attach them to the facial geometry so that changing the muscle parameter values will deform the shape of the facial model. To create smooth animation, one can change the contractions of each muscle resulting in a cha...

Preparing a facial mesh to be animated requires a laborious manual rigging process. The rig specifies how the input animation data deforms the surface and allows artists to manipulate a character. We present a method that automatically rigs a facial mesh based on Radial Basis Functions (RBF) and linear blend skinning approach. Our approach transfers the skinning parameters (feature points and t...

We present a deep learning framework for realtime speech-driven 3D facial animation from just raw waveforms. Our deep neural network directly maps an input sequence of speech audio to a series of micro facial action unit activations and head rotations to drive a 3D blendshape face model. In particular, our deep model is able to learn the latent representations of time-varying contextual informa...

In this paper, we present an approach for real-time speech-driven 3D face animation using neural networks. We first analyze a 3D facial movement sequence of a talking subject and learn a quantitative representation of the facial deformations, called the 3D Motion Units (MUs). A 3D facial deformation can be approximated by a linear combination of the MUs weighted by the MU parameters (MUPs) – th...

Fast, reliable, and marker-free facial expression analysis still remains to be a difficult task in computer vision research. In this paper, the concept of speech-assisted facial expression analysis and synthesis is proposed, which shows that the speech-driven facial animation technique not only can be used for expression synthesis, it also provides useful information for expression analysis. Fr...