Item Consistency Index: An Item-Fit Index for Cognitive Diagnostic Assessment

An item-fit index is a measure of how accurately a set of item responses can be predicted by the test design model. In a diagnostic assessment where items test student mastery on a cognitive model of skills, such an index is needed to determine the alignment between the item responses and skills that each item is designed to measure. In this study, we introduce the Item Consistency Index (ICI), a modification of an existing person-model fit index, for diagnostic assessments. The ICI can be used to evaluate item-model fit on assessments designed with a Qmatrix. Results from both a simulation and real data study are demonstrates the use of our approach. In the simulation study, the ICI was able to identify poor-fitting items under three manipulated conditions: sample size, test length, and proportion of poor-fitting items. In the real-data study, the ICI was used to detect three poor-fitting items for an operational diagnostic assessment in Grade 3 mathematics. Practical implications and future research directions for the ICI are also discussed.