Facial Animation by Synthesis of Captured and Artificial Data

Performer-driven animation has been used with success [1], rst of all to reproduce human body motion. While there are di erent capturing hardware-software systems to map the motion of a performer on the motion of a model of the body or face, little has been done both on the technical and on the theoretical level to support the inventive re-use of captured data. The topic of this paper is an animation editing environment, which can be used to visualize and analyze captured animation data as well as to process the captured data and re-use it several times in di erent ways. The Animation Editor has been designed in the framework of the FASE project, aiming at performer-driven facial animation. The performer data is used to drive both a physically-based 3D realistic model, as well as di erent, cartoonlike 2D faces. We are also experimenting with driving non-human faces with (processed) performer data. The editor is coupled to the models in such a way, that snapshots of the animation or selected parts of it can be seen interwoven with editing. We will discuss animation editing in the context of facial animation. However, most of the ideas could be applied to other animation domains. When animating the human body, techniques used to animate or describe joints the dynamics of motion of physical bodies can be applied. The situation is di erent in the case of facial animation. The `laws' of facial movements are based on too complex and partly unknown physical structures (friction between muscles, precise location and parameters of muscles and tissue, etc.). In addition, there is still very little known about generic and individual characteristics of the dynamics of the face [2]. Hence, both the physical as well as the behavioral laws of facial motion could be, and should be learnt from a large body of empirical captured data. Once generic and individual characteristics of certain facial motions, e.g. expressions, are known, these can be used to process captured data or make facial animations from scratch.