Testing Theories of Risky Decision Making Via Critical Tests

Whereas some people regard models of risky decision making as if they were statistical summaries of data collected for some other purpose, I think of models as theories that can be tested by experiments. I argue that comparing theories by means of global indices of fit is not a fruitful way to evaluate theories of risky decision making. I argue instead for experimental science. That is, test critical properties, which are theorems of one model that are violated by a rival model. Recent studies illustrate how conclusions based on fit can be overturned by critical tests. Elsewhere, I have warned against drawing theoretical conclusions from indices of fit (Birnbaum, 1973, 1974, 2008a): Fit changes under monotonic transformation of the dependent variable and scaling of stimuli. An index of fit depends on experimental design; it depends on parameters and how they are estimated. Different indices can lead to opposite conclusions. A wrong model can achieve a “good” fit, and it can even fit better than the model used to generate the data. I will not add here to this list of problems; instead, I argue in support of traditional science. A theory is a set of statements satisfying five philosophical criteria: (1) it is deductive in that the phenomena to be explained can be derived from the theory; (2) it is meaningful; that is, it can be tested (potentially falsified); (3) predictive: if we knew the theory, in principle, we could have predicted the events to be explained; (4) causal: it specifies in principle how to alter the phenomena via manipulation; and (5) general: premises used in a theory are laws; they are not assumed or denied from case to case. In deduction, when premises are true, conclusions must also be true. However, if a conclusion is assumed (or empirically established), it says nothing about the truth of the premises. Therefore, we cannot “prove” a theory via experiments. However, if implications deduced from a theory are false, we know the theory is false. So we can test a theory by testing its theorems. A test is an opportunity to disprove, but failure to disprove does not prove a theory. The term “model” refers to a special case of a theory that also includes all of the operational definitions and simplifying assumptions needed to apply a theory to a particular paradigm. The classic paradoxes of Allais (1953) are examples of critical tests. These paradoxes lead expected utility (EU) theory into selfcontradiction. They do not require us to estimate any parameters from data, nor do we need to compute an index of fit, because the “paradoxical” behavior, if real, shows that no parameters will work. Models proposed to account for these paradoxes include prospect theory (PT; Kahneman and Tversky, 1979), cumulative prospect theory (CPT) (Tversky and Kahneman, 1992), and the transfer of attention exchange (TAX) model (Birnbaum, 1999). Because people often make different responses when the same choice problem is repeated, it is useful to distinguish instability of preference due to random error from that due to a false theory. The true and error model assumes that different people may have different “true” preferences when presented with a given choice problem, that different choice problems may have different error rates, and that some individuals may have more “noise” in their data than others (Birnbaum, 2008c, Appendix D; Birnbaum and Gutierrez, 2007). This model provides a neutral standard for testing critical properties, such as Allais paradoxes and new paradoxes that distinguish between CPT and TAX. The original version of PT had a number of problems that required a list of “editing rules,” added to excuse the model from potential evidence against it. For example, PT implied that people would violate stochastic dominance in cases where all possible consequences of one gamble are better than the best consequence of the other. So a rule was added to say that people satisfy dominance whenever they detect it, but it did not say when people detect it. CPT solved this problem, because it implies that people always satisfy stochastic dominance, apart from random error. A configural weighting model (Birnbaum and Stegner, 1979), implies that dominance is not always satisfied. A simple version of this model was fit to risky decisions, where it was renamed the TAX model, and a recipe was constructed for choices in which the model predicts a violation (Birnbaum, 1997). Here is an example: