Semi-automatic annotation of eye-tracking recordings in terms of human torso, face and hands

Research on interactive communication increasingly focuses on the role of eye gaze as an important signal in interaction management, reference and grounding (see [1] for an overview). Interlocutors may use eye gaze as a means to take, hold or give the floor in conversation (turn management), to refer to objects or persons in the conversational space (gaze cueing) or to give and elicit feedback (grounding). The use of unobtrusive eye-tracking technology (like eye-tracking glasses or table-top systems) has proven to be an invaluable resource for obtaining detailed information on the distribution of visual attention of multiple participants simultaneously ( [3], [4], [2]). One of the key challenges in the use of mobile eye-tracking technology, however, resides in the processing and annotation of the obtained data stream. An example of an automatic annotation algorithm can be found in our previous work, see [5] for more details. In this paper we further extend our semi-automatic hand annotation algorithm [7], to apply it in the processing of mobile eye-tracking recordings. We integrate the detections of human face, body and hands with gaze data and produce ELAN compatible annotation files.