Modelling rapport in embodied conversational agents

What characterizes the language of people who have known one another for a long time? In the US one thinks of groups of friends, leaning in towards one another, smiling broadly, and finishing one another’s sentences in their eagerness to contribute to the conversation. The details may differ from culture to culture, but the fact of differences between groups of friends and groups of strangers seems universal. And these styles of communication play an important role in identifying people who like and respect us, people we can trust, and people we wish to collaborate with, from the rest of the people in the world. In fact, those differences in communication style, and the feelings they engender, have important consequences for our behavior and actions. For example, we know that children learn better in interactions with teachers and peers that they like, and whom they feel like them. And people who are incapable of reading rapport language or rapport non-verbal behavior or of producing this kind of communicative style – such as people with autism spectrum disorder – not only may be more likely to suffer from depression and loneliness, but also have trouble learning and collaborating. However, we also know that rapport is not always desirable in interaction; in fact, different kinds of interpersonal affiliation behaviors can have different kinds of effects. For instance, the literature suggests that one type of rapport behavior leads survey respondents to answer in ways that they think will please the interviewer, even if these answers are dishonest, while another type of rapport behavior leads respondents to be as truthful as they can, regardless of the social desirability of their responses. Given the importance of interpersonal rapport, it is surprising that so little attention has been paid to implementing notions of long-term rapport in the machines we communicate with. Advances in natural language and speech technology are ensuring that dialogue engines are increasingly emotionally expressive. For the most part, however, those same dialogue engines demonstrate amnesia, beginning every conversation with a user as if it is their first. Even embodied dialogue engines, or Embodied Conversational Agents (ECA), which are increasingly realistic and personality-rich, act like Drew Barrymore in the movie “50 First Dates” – always pleasant, but always meeting somebody for the first time. Because of the importance of notions of rapport, and our belief that rapport can have important consequences for human-machine interaction in domains as diverse as intelligent tutoring systems, assistive technologies, GPS direction-giving, and online survey interviewing, my students and I are currently carrying out a number of different studies to better understand rapport, and to implement it in Embodied Conversational Agents. We are not the only researchers in the field of ECAs to examine this phenomenon. [7] have looked at how ECAs might change their personality and behaviors according to the likes and dislikes of a user, in order to become friends. [1] have reflected on the ways in which computers might need to act in order to maintain relationships with users over time. [5] has looked at the kinds of behaviors that lead users to say that they feel more rapport with an ECA in a first encounter. In our own previous work [2], we looked at the role of small-talk in conversation, and employed our results to implement ECAs that invited increased trust from their users. Our work is characterized, however, by a reliance on models derived from study of humanhuman behavior. That is, we approach the issue by first carrying out a series of studies on rapport in human-human interaction, and using the results to model rapport in ECAs. Importantly, that series of studies attempts to parameterize rapport so that the resultant models can be used across a variety of situations, and so that the functions of rapport are tightly tied to the surface-level behaviors that the agent engages in with a user. Our models join short-term audience design (i.e. grounding phenomena) to long-term audience design (relationship and rapport). And, importantly, our analyses of the data we collect address both language and nonverbal behaviors, and both speech and dis-