Learning and equilibrium as useful approximations: Accuracy of prediction on randomly selected constant sum games

There is a good deal of miscommunication among experimenters and theorists about how to evaluate a theory that can be rejected by sufficient data, but may nevertheless be a useful approximation. A standard experimental design reports whether a general theory can be rejected on an informative test case. This paper, in contrast, reports an experiment designed to meaningfully pose the question: “how good an approximation does a theory provide on average.” It focuses on a class of randomly selected games, and estimates how many pairs of experimental subjects would have to be observed playing a previously unexamined game before the mean of the experimental observations would provide a better prediction than the theory about the behavior of a new pair of subjects playing this game. We call this quantity the model’s equivalent number of observations, and explore its properties. JEL Classification Numbers C72 · C90 This research was supported by a grant from the U.S. National Science Foundation and the USA–Israel Binational Science Foundation. We are very grateful for helpful conversations with David Budescu, Jerry Busemeyer, Gary Chamberlain, Paul Feigin, Dave Krantz, Jack Porter, Tom Wallsten, Wolfgang Viechtbauer, and Richard Zeckhauser. I. Erev Faculty of Industrial Engineering and Management, Technion, Minerva Center for Cognitive Studies, Haifa 32000, Israel E-mail: [email protected] A. E. Roth (B) Department of Economics, Harvard University, Cambridge, MA, USA E-mail: [email protected] R. L. Slonim Department of Economics, Case Western Reserve University, Cleveland, OH, USA A. E. Roth · G. Barron Harvard Business School, Boston, MA, USA