Perception of emotional congruency in multimodal speech synthesis

This working paper experimentally investigates the perception of emotional congruency in multimodal speech synthesis. Therefor two perceptual experiments are described. Experiment 1 is a preliminary test exploring inhowfar subjects are able to identify emotions in synthetic speech as well as in faces presented in short video-clips. Results show that subjects find it easier to recognize emotions in faces than in voices. Experiment 2 investigates the perception of the persuasive power of both emotionally congruent and incongruent audiovisual stimuli by combining the stimuli from the preliminary test. The experiment brought to light that participants judge positive emotionally congruent utterances as significantly more convincing than negative congruent ones. Further positive emotional faces crossed with negative emotional voices were rated as more convincing then negative emotional faces combined with positive emotional voices.