Multimodal Representation of Personified Media with Expressive Strengths of Voice and Gesture

The purpose of this dissertation is to propose and investigate multimodal expression of personified media, with multiple modalities, focusing on the expressive strength of voice and gesture. The results are expected to be used in a practical construction of personified representation. There are many of research works on and products of, personified media such as educational puppets, humanoid robots, virtual agents, communication avatars, etc. which employ a feature that people treat an inanimate being as a human. They aim to create an illusion of personification for virtual agent systems in presenting information, and embodied robots in communication support systems. In those cases, it is desirable to build a shape, internal states, and expressions like humans or living beings as the designer expects. Although detailed personified expressions are partially discussed, their consolidated representation in person-ification is little researched, and most aspects of personified representations are still in research phase. People usually communicate with each other by combining multimodal expressions with verbal and non-verbal cues. Their daily communication includes detailed expressions of vocal tone, facial expression, and motion of the body. They also interpret the other's detailed expressions as a general representation of the person. Consequently, we experimentally discuss the effectiveness of the effectiveness of multimodal representation focusing on vocal and gestural expressions for generally personified media in this dissertation. The naturalness of multimodal representation does not necessarily correspond to the naturalness in each modality. This dissertation therefore describes the investigation on both expressions