Automated Content Scoring of Spoken Responses in an Assessment for Teachers of English

This paper presents and evaluates approaches to automatically score the content correctness of spoken responses in a new language test for teachers of English as a foreign language who are non-native speakers of English. Most existing tests of English spoken proficiency elicit responses that are either very constrained (e.g., reading a passage aloud) or are of a predominantly spontaneous nature (e.g., stating an opinion on an issue). However, the assessment discussed in this paper focuses on essential speaking skills that English teachers need in order to be effective communicators in their classrooms and elicits mostly responses that fall in between these extremes and are moderately predictable. In order to automatically score the content accuracy of these spoken responses, we propose three categories of robust features, inspired from flexible text matching, n-grams, as well as string edit distance metrics. The experimental results indicate that even based on speech recognizer output, most of the feature correlations with human expert rater scores are in the range of r = 0.4 to r = 0.5, and further, that a scoring model for predicting human rater proficiency scores that includes our content features can significantly outperform a baseline without these features (r = 0.56 vs. r = 0.33).