Integrating Perceptual and Cognitive Modeling for Adaptive and Intelligent Human-Computer Interaction

This paper describes technology and tools for Intelligent HCI (IHCI) where human cognitive, perceptual, motor, and affective factors are modeled and used to adapt the H C interface. Intelligent HCI emphasizes that human behavior encompasses both apparent human behavior and the hidden mental state behind behavioral performance. IHCI expands on the interpretation of human activities, known as W4 (what, where, when, who). While W4 addresses only the apparent perceptual aspect of human behavior, the W5+ technology for IHCI described in this paper addresses also the why and how questions whose solution requires recognizing and processing around specific cognitive states. IHCI integrates parsing and interpretation of nonverbal information with a computational cognitive model of the user, which, in turn, feeds into processes that adapt the interface to enhance operator performance and provide for rational decision-making. The technology proposed is based on a general fourstage, interactive framework, which moves from parsing the raw sensory-motor input, to interpreting the user’s motions and emotions, to building an understanding of the user’s current cognitive state. It then diagnoses various problems in the situation and adapts the interface appropriately. The interactive component of the system improves processing at each stage. Examples of perceptual, behavioral and cognitive tools are described throughout the paper. Adaptive and Intelligent HCI are important for novel applications of computing including ubiquitous and human-centered computing.