Looking at Sounds: Neural Mechanisms in the Primate Brain

When you hear a salient sound, it is natural to look at it to find out what is happening. Orienting the eyes to look at sounds is essential to our ability to identify and understand the events occurring in our environment. This behavior involves both sensorimotor and multisensory integration: A sound elicits a movement of the visual sense organ, the eye, to bring the source of the sound under visual scrutiny. How are auditory signals converted into oculomotor commands? This chapter describes our recent work concerning the necessary computational steps between sound and eye movement, and how they may be implemented in neural populations in the primate brain. In principle, the brain must determine the location of the sound, encode that location in a reference frame and format that allows for convergence with visual signals onto a common motor pathway, and create a suitable timevarying signal in the extraocular muscles to move the eyes. In practice, it is not clear exactly how these computations unfold. Several specific hurdles must be overcome. First, auditory and visual signals arise in different reference frames. Binaural and spectral cues provide information about where a sound is located, but only with respect to the head and ears, not the eyes. In contrast, visual information is intrinsically eye centered: The pattern of illumination of the retina depends on the locations of objects in the visual scene with respect to the direction of gaze. These two reference frames vary in their relationship to each other depending on the orbital position of the eyes (Fig. 15.1). This discrepancy in reference frame should be resolved prior to or as part of the convergence of visual and auditory signals onto a common oculomotor pathway. A second computational hurdle is that visual and auditory signals are not necessarily encoded in the same format. From the retina on, neurons in the early visual pathway have receptive fields that tile the visual scene and produce a ‘‘place code’’ for stimulus location (Fig. 15.2). In contrast, the binaural computations performed in the auditory pathway do not necessarily produce receptive fields. If they do not, then there may be a discrepancy in the coding format of visual and auditory signals. Ultimately, either visual or auditory or both signals must undergo a transformation into a reference frame and a coding format that are similar to each other and appropriate for accessing the oculomotor pathway. We will begin by describing the evidence concerning the reference frame of auditory signals as they progress from auditory to multimodal and oculomotor areas before turning to coding format and some computational analyses that shed light on the neural algorithms that may be at play in this process.