Error Rate Bounds in Crowdsourcing Models

Crowdsourcing is an effective tool for human-powered computation on many tasks challenging for computers. In this paper, we provide finite-sample exponential bounds on the error rate (in probability and in expectation) of hyperplane binary labeling rules under the Dawid-Skene crowdsourcing model. The bounds can be applied to analyze many common prediction methods, including the majority voting and weighted majority voting. These bound results could be useful for controlling the error rate and designing better algorithms. We show that the oracle Maximum A Posterior (MAP) rule approximately optimizes our upper bound on the mean error rate for any hyperplane binary labeling rule, and propose a simple data-driven weighted majority voting (WMV) rule (called one-step WMV) that attempts to approximate the oracle MAP and has a provable theoretical guarantee on the error rate. Moreover, we use simulated and real data to demonstrate that the data-driven EM-MAP rule is a good approximation to the oracle MAP rule, and to demonstrate that the mean error rate of the data-driven EM-MAP rule is also bounded by the mean error rate bound of the oracle MAP rule with estimated parameters plugging into the bound. 1. Introduction. There are many tasks that can be easily carried out by people but tend to be hard for computers, e.g. image annotation and visual design. When these tasks require large scale data processing, outsourcing them to experts or well-trained people may be too expensive. Crowdsourcing has recently emerged as a powerful alternative. It outsources tasks to a distributed group of people (usually called workers) who might be inexperienced in these tasks. However, if we can appropriately aggregate the outputs from a crowd, the aggregated results could be as good as the results by an The flaws of crowdsourcing are apparent. Each worker is paid purely based on how many tasks that he/she has completed (for example, one cent for labeling one image). No ground truth is available to evaluate how well he/she has performed in the tasks. So some workers may randomly submit answers independent of the questions when the tasks assigned to them