The Teleface project - disability, feasibility and intelligibility

The Teleface project, a project that aims at evaluating the possibilities for a telephone communication aid for the hard of hearing, is presented as well as the different parts of the project: audio-visual speech synthesis, visual speech measurement and multimodal speech intelligibility studies. Some results from the first test series are presented. Introduction It is well known that visual information obtained by speechreading and interpretation of body gestures improves perception of speech, and especially so in a noisy environment. Sumby & Pollack (1954) has described the amount of visual information from lip-reading as a function of the signal to noise level. The visual information is even more important to persons with a hearing loss. The Teleface project at KTH focuses on the usage of multimodal speech technology for hearing impaired people. The first phase of the project aims at evaluating the increased intelligibility hearing impaired people might experience from an auditory signal if it is supplemented by a synthesized face. The intelligibility study will be the main focus of this paper. In the second phase, which has not yet been initiated, we will try to implement a demonstrator of a telephone communication aid for the hard of hearing. This device will generate a synthetic face that articulates in synchrony with the telephone speech. Control parameters for the face will be determinedby analysis of the telephone speech signal. Such a device would give the user the benefit of speech reading during any telephone conversation. (In contrast to a video telephone solution, which requires both parties to be equipped with compatible video telephone hardware.) This visual hearing aid could be implemented as software running on a PC, or as a dedicated standalone unit (the “Teleface” unit). Other applications of our research includes speech based user-interfaces and spoken dialog systems. Talking animated agents, using visual speech synthesis, has been employed in recent spoken dialog system research projects, such as Waxholm (Bertenstam et al, 1995) and the Olga system (Beskow, Elenius & McGlashan, 1997). Even if a hearing impaired person is not the obvious user of a spoken dialog system, a synthetic face could actually make it possible for hearing impaired people to take advantage of the increased flexibility and user friendliness expected from such systems in the future. Synthetic visual speech The visual speech synthesis is done using three-dimensional parameterised facial models, controlled by a text-to-speech rule system (Beskow 1995). Within the project, this control scheme will be refined, using data obtained by optical measurements of a real speaker, in order to improve the naturalness of the facial movements (Öhman, 1997). In the intelligibility study we use two different face models. One is an extended version of the Parke model (Beskow