Evaluating the Wisdom of Crowds

Estimating and forecasting are difficult tasks. This is true whether the activity requires the determination of uncertain future event outcomes, or whether the estimation effort is complex in itself and based on insufficient information. Consequently, such tasks are frequently assigned to experts. Surprisingly, recent research suggests that collectives of non-experts can outperform individual experts, as long as certain conditions are met. The resulting capability has been described as collective intelligence or “wisdom of crowds”. Our research explores this collective intelligence, for real and simulated crowds. An empirical test demonstrates that even a relatively small crowd of 30 subjects can demonstrate expertlike performance. A further investigation through simulation shows that the performance of a collective predictably compares to that of an expert, with the expert outperforming small crowds but being outperformed by large collectives. The relationship between performance and log of collective size follows a linear function.