Usability of MultiPoint

MultiPoint is a speech and pen user interface for building presentations, implemented as an add-on to Microsoft PowerPoint. We compared users’ satisfaction and performance between building presentations with MultiPoint and with PowerPoint. We also compared participants’ performance and reactions between using Wizard of Oz (WOz) speech recognition and computer speech recognition. In aggregate, most of the participants ranked the naturalness of MultiPoint speech commands high regardless of whether they were using WOz recognition or computer speech recognition. Six participants with WOz recognition completed tasks in about the same time and with about the same number of errors and with the same satisfaction, as they did using PowerPoint alone. Six participants with computer speech recognition took twice the time and committed four times as many errors and had significantly lower satisfaction, as they did using PowerPoint. INTRODUCTION Pen and speech interfaces are likely to become more important in the future with increasing use of tablet computers and other handheld devices. In an attempt to expand on the tasks where multimodal input might be more appropriate [COHEN, OVIATT], we undertook the task of building a multimodal interface for building presentations, a common desktop task, and compared it to a regular graphical user interface with keyboard and mouse. Our interviews with professionals in industry led us to the conclusion that building presentations is a task potentially well-suited to a multimodal interface on a tablet computer. Out interviewees often draw the first draft of their slides; we designed MultiPoint as a tool for creating the first draft of a set of slides on a pen computer. MULTIPOINT IMPLEMENTATION Multipoint is implemented as an add-on to Microsoft PowerPoint and uses SRI’s Open Agent Architecture (OAA) [MORAN]. It communicates to PowerPoint via Visual Basic for Applications (VBA). OAA facilitates synchronous and asynchronous communication managed by a central facilitator among different recognition agents in the system. This provides the facilities for pen and speech recognition in MultiPoint. MultiPoint includes a subset of PowerPoint functionality; there are approximately 60 multimodal commands that can be used to draw and dictate slides. A user starts with the freeform drawing tool and draws a word or a shape on the slide. He simultaneously speaks speech commands to act on the freeform object. As an example, the participant draws a rectangular shape, says, “make a rectangle,” and the sketch is converted into recognized form (Figure 1). MultiPoint confirms the command audibly by repeating the recognized command via text to speech. Speech commands in MultiPoint are particularly useful for 1. guiding recognition of sketched items 2. changing properties of existing objects 3. setting animations of existing objects Some example commands used in the user study include: • “add title” • “add bullet” converts a sketched rectangle into neatened form Figure 2. A partici MultiPoint on a tab “Make a Rectangle” Engaging the audience • Eye contact • Speak loudly • Hand gestures User draws a rectangle Engaging the audience • Eye contact • Speak loudly • Hand gestures Neatened rectangle appears Figure 1. The MultiPoint command “make a rectangle” • “make a rectangle” • “make a triangle” • “color shape red” • “fly from right” • “delete” • “undo” By explicitly switching into “dictation mode” a participant can also dictate text. Text can also be inputted via a handwriting recognizer agent. pant using let computer METHOD In the user study, 12 paid student participants were asked to copy four slides using MultiPoint and PowerPoint (Figure 4). For MultiPoint, each used a Fujitsu Stylistic 4000 pen computer and a headset microphone (Figure 2). For PowerPoint each used a laptop computer and an external mouse. Participants were also allowed to create a freeform slide about their jobs after the copy tasks. Each participant was trained on the MultiPoint commands and the PowerPoint operations that they could use to draw the slides. The copy tasks were presented in randomized order. One-half (six) performed their tasks with WOz speech recognition, and the other half (six) of the participants performed the tasks with computer speech recognition from IBM ViaVoice. The User Interface community typically operationalizes “performance” as the time, steps, and errors for completion of certain tasks. Each task was timed, and the number of steps and errors were counted. Steps included “adding a square,” “changing the color of a triangle,” and were counted at the same granularity in both MultiPoint and PowerPoint. Errors included mistakes, undos, and speech misrecognitions. Participants were surveyed about their PowerPoint experience before the study and about their evaluation of the system after the study. RESULTS Time, steps, and errors were the three metrics most important to us in the comparison of MultiPoint and PowerPoint. Satisfaction values from the survey were also important. Within Subject Analysis A set of error bar plots with 95% confidence intervals (Figure 5, Figure 6, Figure 7) show the mean value of these metrics for each of the four tasks in the two groups of participants, those who ran MultiPoint with Wizard of Oz speech recognition and those who ran MultiPoint with PowerPoint speech recognition. The Wizard of Oz speech recognition cases generally overlap with PowerPoint, suggesting a lack of statistically significant differences between the two. The Computer speech recognition cases exceed the PowerPoint values by some amount for some of the tasks. Rather than compare each task individually, we totaled values and performed analysis on the sum of the time, sum of the steps, and sum of the errors for each of participants. In what follows, we consider the MultiPoint and PowerPoint performance Wizard of Oz speech recognition group and the Computer Speech recognition group separately. This allows us to do a within subjects analysis of the performance and satisfaction metrics that were collected. In the WOz speech recognition tests, there were no statistically significant differences in the total time (MD=21, SD=210), t(5)=0.243, p=0.818 (Figure 8) or the total steps for completion (MD=-2.2, SD=31), t(5)=-0.173, p=0.870 (Figure 9) or the total errors (MD=-8.2, SD=12), t(5)=-1.70, p=0.149 (Figure 10) between MultiPoint and PowerPoint. (There were no speech recognition errors in the WOz SR condition.) This result means that MultiPoint performs as well as PowerPoint, even for participants experienced with PowerPoint and new to MultiPoint. For computer speech recognition, there were statistically significant differences in total time, total steps, and total errors. MultiPoint users took 215% of the time (M=409, SD=163), t(5)=6.12, p=0.0017 (Figure 8), 148% of the steps (M=33, SD=13), t(5)=6.11, p=0.0017 (Figure 9), and 427% of the errors (M=27, SD=13), t(5)=5.11, p=0.0037 (Figure 10), versus PowerPoint. From observation during the test, it was clear that most of the additional MultiPoint steps versus PowerPoint steps were from speech recognition errors and correcting those errors. Reducing the error rate might make performance more similar to WOz recognition. For WOz speech recognition participants, there were no statistically significant differences between MultiPoint and PowerPoint in participant rankings, scaled 1-5, of ease of use (MD=-0.33, SD=0.82), t(5)=-1, p=0.36 (Figure 11), quickness (MD=-0.33, SD=1.63), t(5)=-0.5, p=0.638 (Figure 12), and naturalness (MD=0.17, SD=1.33), t(5)=0.307, p=0.771 (Figure 13). All three beginners to PowerPoint ranked MultiPoint higher for ease of use, noting specifically its simplicity and pleasantness. Participants using the computer speech recognizer were speech recognition conditions and two tools.