Evaluating Crowdsourcing through Amazon Mechanical Turk as a Technique for Conducting Music Perception Experiments

Online crowdsourcing marketplaces, such as the Amazon Mechanical Turk, provide an environment for cost-effective crowdsourcing on a massive scale, leveraging human intelligence, expertise, and judgment. While the Mechanical Turk is typically used by businesses to clean data, categorize items, and moderate content, the scientific community, too, has begun experimenting with it to conduct academic research. In this paper, we evaluate crowdsourcing as a technique for conducting music perception experiments by first describing how principles of experimental design can be implemented on the Mechanical Turk. Then, we discuss the pros and cons of online crowdsourcing with respect to subject demography, answer quality, recruitment cost, and ethical concerns. Finally, we address audio-specific factors relevant to researchers in the field of music perception and cognition. The goal of this review is to offer practical guidelines for designing experiments that best leverage the benefits and overcome the challenges of employing crowdsourcing as a research methodology.