Integrating Facial, Gesture, and Posture Emotion Expression for a 3d Virtual Agent

Facial expressions, gestures, and body postures can portray emotions in a non-verbal way. These methods are frequently employed by actors in theatrical plays or in movies, or even by virtual characters such as those found in computer games, animated storybooks, and website e-assistants. Signals for emotion expressions (“cues”), such as a raised fist and narrowing of the eyes, substantially influence the viewers’ assumptions on the emotional state of the person portraying it. To enable humans to recognize emotions of virtual characters, the characters’ cues must be portrayed according to the human counterparts. In this study, we first look at existing systems for synthesizing cues for facial expressions, gestures and body postures. This is followed by examining the emotion recognition problems that arise from utilizing the various systems. Lastly, the systems are integrated together and the implications that arise from the integration are analyzed. The study showed that hand gestures aided in the emotion recognition rate for postures which others [Coulson 2004] had previously assumed as unimportant. Additionally, it was discovered that the emotion recognition rate using gestures can be greatly improved when emblematic actions are combined with functional actions. This study also confirmed our assumption that an integrated system covering facial expression, gesture and body postures is capable of enhancing the emotion recognition rate beyond the rates of the single systems.