Change of Perception of Emotions in Multimodal and Crosslinguistic Setttings

This paper adresses two questions. The first question is if perception of emotional prosody changes when visual stimuli are present. It has already been shown by others that the perception of vowels and consonants is augmented or changed when visual information is present.The question is then if perception of such categories as emotions, mainly expressed by prosody, are also affected by multimodal stimuli. The second question concerns the cross-linguistic perception of multimodal stimuli. Earlier cross-cultural studies indicate that facial expression of emotions is more universal than prosody is. Cross-linguistic interpretation of emotions could then be more successful multimodally than only vocally. The specific questions asked in the present study are to what degree Spanish and Swedish listeners can interpret Spanish emotional prosody, and whether simultaneously presented faces, expressing the same emotions, change or improve the interpretation. Audio recordings of Spanish emotional expressions were presented to Spanish and Swedish listeners, in two experimental settings. In the first setting the listeners only attended to prosody, in the second one they also saw a face, expressing different emotions. The results indicate that intra-linguistic as well as cross-linguistic perception of emotional prosody is improved by visual stimuli. Crosslinguistic interpretation of prosody is more poorly accomplished than inter-linguistic, but seems to be greatly augmented by multimodal stimuli. F0 analysis of the stimuli is made and discussed in relation to the results of he perceptions of the stimuli of the different language groups. The results are also analysed in terms of gender of listeners.