Anthropocentric Video Analysis: Tools and Applications

During the last two decades, we have witnessed an increasing research interest towards what one could call anthropocentric video analysis, namely, algorithms that aim to extract, describe, and organize information regarding the basic element of most videos: humans. This diverse group of algorithms processes videos from various sources (movies, home videos, TV programmes, surveillance videos, etc.) and extracts a wealth of useful information. A large cluster of algorithms target information related to the state or state transitions of individuals: presence and position/posture through face or body detection, body or body parts tracking and posture estimation; identity by means of face recogni-tion/verification, full-body recognition, gait analysis, and so forth; emotional state through facial expression, body gesture , and/or posture analysis; performed actions or activities; and behavior through spatio-temporal analysis of various behavioral cues including facial/head/hand/body gestures and postures. Another smaller group of techniques focuses on detecting or recognizing interactions or communication modes by means of visual speech recognition, dialogue detection, social signals recognition such as head nods and gaze exchanges, recognition of activities or events in multiple-person environments (e.g., event analysis in sport videos or crowd-scene analysis, etc.). Finally, a number of techniques aim at deriving information regarding physical characteristics of humans, mainly in the form of 3D head or full-body models. The interest of the scientific community for anthro-pocentric video analysis stems from the fact that the extracted information can be utilised in various important applications. First of all, it can be used to devise intuitive and natural paradigms of man-machine interaction, for example, through gesture-based interfaces, visual (or audiovisual) speech recognition, interfaces that understand and adapt to the emotional state of users, and interfaces between virtual characters and human users, which are governed by the same social rules as the human-human interaction. In the same wavelength, but in a considerably broader scope, anthropocentric video analysis techniques are some of the enabling technologies for the so-called ubiquitous computing trend (also known as pervasive computing or ambient intelligence) where a large number of small (or embedded), interconnected, and clever computing devices and sensors cooperate to assist people in their everyday life in an unobtrusive and natural way. An intelligent living space, that controls lighting, music, temperature, and home appliances according to the inhabitants' mood, location, habits, and behavioral patterns indicating their intention, is frequently used as an example of this trend. Moreover, techniques like person detection, tracking, recognition or verification, and …