Experimental Validation of TranScribe Prototype Design

In this paper we describe an experiment to compare a new transcription design prototype to an existing commercial application. In a between-subjects design, participants transcribed short interview segments using both applications, and they were compared with regards to quantitative efficiency measures such as time taken, and also on subjective user ratings assessed using Likert scales on an exit questionnaire. Discouragingly, most measures found no significant improvements of our prototype on the commercial product, although user satisfaction was significantly higher in the prototype. Over longer transcription periods and with additional user-supporting features, we expect our application to yield greater efficiency and even higher satisfaction, although some design features such as shortcut key placement will have to be improved. INTRODUCTION We have been interested in developing a software transcription tool to help occasional transcribers to produce accurate transcriptions efficiently. A heuristic evaluation gave us the confidence that we had identified the major issues with our design, and we created a functional prototype in C#, which would allow us to evaluate our design not only through expert reviews, but in a rigorous quantitative study. We traced our improvement decisions back to the beginning of the process, when we gathered user requirements on a commercial transcription tool called Express Scribe. The problems identified in that requirements study served to guide us in designing TranScribe, and so the next step in the design process would intuitively be to use Express Scribe as the benchmark for our prototype in terms of efficiency, user satisfaction and on the principal design questions we have been faced with. The study described below was designed as a direct comparison between the two transcription applications, which has the obvious drawback of comparing a fully tested commercial application to a functional prototype coded over the course of a week. Nevertheless, the prototype of TranScribe (see Appendix A) implemented the same audio playback and text editing features that were used in Express Scribe, and was created to minimize confounds in the experiment. METHOD Participants Six adults (2 female, 4 male) with normal or corrected-tonormal visual acuity were recruited for the experiment. Design In this experiment, our prototype was compared directly to Express Scribe, the commercial transcription application that was evaluated for usability issues at the beginning of this project. In a repeated-measures design, each participant was asked to transcribe two one-minute segments of the same interview, once using Express Scribe, and once using TranScribe. The two segments appeared in the same order, but the transcription applications were counterbalanced to minimise primacy and fatigue effects. Each transcription was timed as a measure of efficiency. In our previous experiments, it was determined that users resort to the mouse when they cannot find a way to accomplish a task efficiently with the keyboard, which is why the number of mouse clicks was recorded for each condition as an indication of interrupted typing flow. Then, a 7-point Likert scale questionnaire obtained subjective measures of the ease and enjoyment of using each application, their hotkey placements, and the participant’s overall preference between the two (see Appendix A). Since TranScribe was created specifically to address problems encountered with Express Scribe, it was hypothesized that transcriptions done in TranScribe would be faster than those done in Express Scribe, that TranScribe would score higher in overall usability and shortcut key placement, and that it would be preferred by participants overall. Apparatus Experiments were conducted on a 15.4'' widescreen portable computer with a screen resolution of 1280x800 using an external keyboard and mouse. Green painter's tape was applied to the function keys which served as shortcuts to Express Scribe and TranScribe and they were labeled appropriately. Participants were fitted with a headset and the volume level was adjusted individually during the practice transcription. Both applications allowed for audio playback at two different speeds (50% and 100%) while preserving pitch, and had fine-grained rewind and fast forward functionality. They also had different shortcut key maps (See Fig. 1) Transcriptions were timed to the nearest second with the stopwatch function on a portable media player and the number of mouse clicks were counted by one of the experimenters. Figure 1. Express Scribe (above) and TranScribe (below) playback shortcuts. Procedure Participants were seated in front of the experimental PC in a quiet conference room. They were given an instructions form and asked to sign an informed consent (see Appendix B). An experimenter described the steps to the experiment, and they were introduced to the first transcription interface. They were allowed to familiarize themselves with the playback and the flow of the interface on a practice audio file from a different interview. Then they completed the transcription. They were asked to identify speakers by A and B, and to produce a transcription of good quality efficiently. The same steps were repeated for the other transcription application.