9 Auditory ‐ Visual Speech Processing Something Doesn ’ t Add Up ErIc VATIkIoTIS ‐

The multimodal production and multisensory perception of speech have received much research attention in the past 60 years since Sumby and Pollack’s landmark demonstration that being able to see a talker’s face in noisy acoustic conditions dramatically improves speech intelligibility (Sumby and Pollack 1954). Myriad studies have pursued various conceptual lines about the production and processing of multisensory information in the context of diverse tasks applied to clinical populations and hordes of undergraduate psychology students, and in technical applications for multimedia and speech technology. Previously, we have reviewed the progress in auditory‐visual speech processing, particularly with respect to the production and perception of time‐varying speech behavior (Munhall and Vatikiotis‐Bateson 1998, 2004; Vatikiotis‐Bateson and Munhall 2012a, 2012b). In this chapter,1 we examine what has been learned about auditory‐ visual speech processing (AVSP) from the potentially disturbing perspective that we still do not have a cogent story for how the visual enhancement of speech intelligibility works. Examining the neural underpinnings of AVSP is, of course, a promising and increasingly well‐worn path toward working out a suitable story. However, before turning to neurophysiology to account for behavior, we think it worthwhile to critically review what we have and have not learned from behavioral studies of the production and perception of multimodal speech. In particular, the research questions that have been asked were based on premises and assumptions about language and cognition that may need to be rethought before the observed results can begin to make sense, and should be reexamined 9 Auditory‐Visual Speech Processing Something Doesn’t Add Up