Goal Orientation, Self-Efficacy, and "Online Measures" in Intelligent Tutoring Systems

While goal orientation and related factors like learner selfefficacy are of great interest to learning science researchers, some voice concerns regarding the measurement of such factors using self-report questionnaires. To address these concerns, recent work has explored the use of behavioral indicators like hint-seeking and glossary use in intelligent tutoring systems like Carnegie Learning’s Cognitive Tutor (CT) as alternative, “online” measures of goal orientation. We re-examined this approach by measuring 273 CT users’ achievement goals and self-efficacy judgments via embedded questionnaires and their hint-seeking and glossary use via log data. Using graphical causal models and linear structural equation models to observe structural relationships among goal orientations, self-efficacy, behaviors, and learning outcomes, we found that tracing orientations via “online measures” is more nuanced than perhaps previously appreciated. We describe complex relations observed in the model among motivations, behaviors, and outcomes and discuss the implications for the online measurement of motivation.