Automated Classification of Complex Mental States from Head and Facial Displays in Video

This paper presents a system for the automated inference of complex mental states from observed facial expressions and head gestures in video. Head actions are identified from image-based pose estimation. Facial actions, including asymmetric ones, are extracted from motion, color and shape parameters. A multi-level dynamic Bayesian network classifier models complex mental states as a sequence of interacting facial and head displays. We evaluate the recognition of six complex mental states groups. Our experiments demonstrate the effectiveness of our approach to fully automated, head motion resilient recognition of displays representing complex mental states.