Assessing subjective vs objective cognitive artifact usage

For decades, IS researchers have discussed the reliability of subjective measurements to assess actual artifact usage. Especially in experimental settings, as in the design science context for instance, the participants’ usage data of the evaluated artifact represents an important point of analysis. However, collecting objective usage data, (i.e. logfiles) is often not feasible depending on the artifact. In this paper, we present the theoretical grounding of collecting cognitive artifact usage data using eye-tracking technology. Grounded in immediacy and eye-mind assumption the participants’ artifact fixations are used as objective usage measurement. The question remains if in comparison, the collection of subjective (e.g. perceptual) usage data is sufficient and reliable for such experiments. The results of our comparative analysis indicate that researchers could use subjective measurements when comparing different artifact designs and should rely on objective measurements when testing the effect of an artifact compared to a control group without artifacts.