Facial Feature Based Head Tracking and Pose Estimation

Main purpose of this thesis is to study a facial feature based human head pose estimation. Especially the facial feature extraction methods suitable for a real-time system are considered. Based on the literature survey, an experimental head tracking and pose estimation system is implemented. An efficient method based on skin detection, gray-scale morphology and a geometrical face model are applied to detect roughly the face region and extract facial features. The facial features extracted include the eyes, mouth, nostrils and eyebrows. The system automatically initializes whenever the face of a user is detected. After successful initialization, the tracker is started. The extended Kalman filtering (EKF) framework is utilized for tracking and estimation of the 3-D pose of the moving head in an image sequence. During tracking the correspondences between a rigid head model and three extracted facial features (eyes and mouth) are used to solve the pose. To demonstrate the practicality of the approach the user’s gaze direction is detected and applied to a cursor control on the computer screen. Experiments with real image sequences and synthetic data were carried out. Tests with real sequences showed that the system is able to extract facial features reliably from a near frontal view of the face under proper lighting conditions. Furthermore, high real-time performance was achieved. Pose estimation and gaze detection accuracies were tested with synthetic data and good preliminary results were obtained. The fast implementation proves the basic premise that the proposed approach can be applied in platforms like mobile devices where high computational resources are not available. However, there are some restrictions in the current implementation. A user with glasses, occlusion and shadows will cause problems for the system. In future work the observed problems will be considered more closely.