Methods for Classifying Errors on the Raven's Standard Progressive Matrices Test

Although many psychometric tests, like Raven’s Progressive Matrices, are commonly evaluated according to total score, additional variables can lend insight into the underlying cognitive processes. We examine conceptual errors on the Raven’s Standard Progressive Matrices (SPM) test. We present a complete classification of error types on the SPM using a two-kind coding scheme, yielding ≥ 95% inter-rater reliability. We also examine how to extract error data from a computational model, and we present a method for measuring errors through systematic ablation to create a “population” of models whose performance can be examined as a group. We present a preliminary analysis of error patterns on the SPM from typically developing individuals, individuals diagnosed with autism, and a computational model called ASTI. We discuss what the error patterns suggest regarding cognition on the SPM and routes towards improving the ASTI model.