Preference Heterogeneity in Experiments: Comparing the Field and Lab

Economists recognize that preferences can differ across individuals. We examine the strengths and weaknesses of lab and field experiments to detect differences in preferences that are associated with standard, observable characteristics of the individual. We consider preferences over risk and time, two fundamental concepts of economics. Our results provide striking evidence that there are good reasons to conduct field experiments. The lab fails to detect preference heterogeneity that is present in the field, obviously due to the demographic homogeneity of the lab. There are also differences in treatment effects measured in the lab and the field that can be traced to interactions between treatment and demographic effects. These can only be detected and controlled for properly in the field data. Thus one cannot simply claim, without additional empirical argument or assumption, that treatment effects estimated in the lab are reliable. † Centre for Economic and Business Research, Copenhagen, Denmark (Andersen and Lau) and Department of Economics, College of Business Administration, University of Central Florida, USA (Harrison and Rutström). E-mail contacts: [email protected], [email protected], [email protected] and [email protected]. Rutström thanks the U.S. National Science Foundation for research support under grants NSF/IIS 9817518, NSF/MRI 9871019 and NSF/POWRE 9973669, and Harrison and Lau thank the Danish Social Science Research Council for research support under project #24-02-0124. Supporting data and statistical code are stored in the ExLab Digital Archive at http://exlab.bus.ucf.edu 1 For example, assume a policy that is predicted to result in either a zero or a positive (2,000 DKK, for example) effect on the income of the average Danish household, with probabilities 1⁄4 and 3⁄4, respectively. The certainty equivalent of this policy, assuming a risk coefficient in the neighborhood of those we estimate for the field, is about 40% lower than the expected value of 1,500 DKK. Even if we assumed a 90% confidence in the policy prediction instead of 75% confidence, we would be off by almost 20% if we maintained risk neutrality. -1Economists recognize that preferences can differ across individuals. We examine the characterization of preference heterogeneity through controlled experiments. Laboratory experiments have become a generally accepted methodology in mainstream economics, but we are concerned that for some tasks the convenience samples drawn from the relatively homogeneous population on university campuses may lead to misleading inferences. This is particularly likely to be a problem for exercises that entail the explicit elicitation of subjective preferences. If the estimated preference parameters are to be used in policy analysis, the lab findings may be irrelevant since much of the heterogeneity of the broader population is not being represented. We focus on the strengths and weaknesses of lab and field experiments to detect differences in preferences, particularly those differences that are associated with standard, observable characteristics of the individual. The importance for policy analysis of representing risk and time preferences appropriately, rather than relying on convenience assumptions such as risk neutrality and market interest rates, is demonstrated in Harrison, Lau and Rutström [2004]. They show that a policy-maker could overstate the welfare gains to households by as much as 42% in realistic tax policy simulations by ignoring the risk aversion of the average Dane.1 One important finding is that we are able to detect much richer preference heterogeneity when we sample from the field, as compared to convenience samples drawn from college students in our lab experiments. We say “detect” because we do not claim that the heterogeneity is missing in the sample used in the lab, just that the range of variation in characteristics is understandably smaller. The convenience samples are more homogeneous with respect to many observable characteristics, such as age and education level. Thus one would expect that it would be harder to detect differences statistically, even if they were present. Since we have the benefit of comparable lab and field experiments, we can claim that there is preference heterogeneity in the field, and so we know that we are missing something in the lab. We therefore conclude that the lab might not be the