Widening the Evaluation Net

Intelligent Virtual Agent (IVA) systems are notoriously difficult to evaluate, particularly due to the subjectivity involved. From the various efforts to develop standard evaluation schemes for IVA systems the scheme proposed by Isbister & Doyle, which evaluates systems across five categories, seems particularly appropriate. To examine how these categories are being used, the evaluations presented in the proceedings of IVA ’07 and IVA ’08 are summarised and the extent to which the five categories in the Isbister & Doyle scheme are used is highlighted. 1 IVA Evaluations and IVA ’08 and IVA ’09 As Intelligent Virtual Agent (IVA) research has matured, evaluation has become more important. However, evaluation of IVA systems is notoriously difficult as there are a whole range of issues that must be considered (e.g. are the behaviours of agents believable?, are agents socially capable?, does the system run efficiently in real-time? ), and that these issues tend to be quite subjective. However, without good evaluations it is very difficult to compare competing systems and track the development of the field as a whole. Fortunately, there are a number of proposed standard evaluation schemes for IVA research. One scheme that seems particularly useful was proposed by Isbister & Doyle [1] for evaluating pedagogical conversational agents which evaluates systems under five categories: Believability, Social Interface, Application Domains, Agency & Computational Issues, and Production. To examine the state-of-the-art in evaluation in IVA research, the evaluations described in the proceedings of IVA ’07 [2] and IVA ’08 [3] were summarised. Each full paper published (31 and 45 in IVA ’07 and IVA ’08 respectively) was examined, and the evaluations described were categorised under the 5 categories in the Isbister & Doyle scheme. Papers for which evaluation is simply inappropriate are placed under the category N/A. Finally, those papers that do not describe any evaluations are placed in the category None. Figure 1 shows first how many of the papers in each year evaluate under each of the categories in the scheme, and the N/A and None categories; together with histograms of how many of the categories are covered in the evaluations presented each year. 2 Conclusions & Future Work The points to notice from the graphs in figure 1 are: there are a large number of papers in which no evaluation is described; it is clear that some of the evaluation