Cynicism, Conspiracies, and Contemporaneous Conditions Moderating Experimental Treatment Effects

Survey experiments are widely used in political science. Among other things, they promise internally valid estimates of a wide variety of causal effects. One specific threat to this assumption is that participants might bring other information based on real-world conditions to these experiments that could mediate posited experimental effects. We investigate this possibility by comparing the results of an experiment concerning conspiracy theories and trust in government conducted at a fairly routine time to results from the same experiment collected in a high-scandal period. We find that the results from the high-scandal period deviate from the original results in ways theoretically consistent with the real world affecting the experiment. Substantively, our findings show that cynicism and conspiracy theories are linked in a vicious cycle in which macro conditions affect conspiratorial thinking. More generally, these findings, especially when combined with previous research, suggest an important threat to the interpretation of at least some survey experiments’ results. 6,141 words ∗Authors names are listed alphabetically. Working paper prepared for the 2013 APSA meetings. We’d like to thank Brendan Nyhan and Doug Kriner for insightful comments. In recent years, survey experiments have become increasingly popular in political science research. Their main promise is internally valid identification of direct causal effects. While survey experiments have enabled many important contributions to a variety of literatures, some prominent recent scholarship raises important questions about how cleanly these experiments can parse effects and how other stimuli (including experiments) can contaminate such designs and confound estimation and interpretation (Gaines, Kuklinski and Quirk, 2007; Druckman and Leeper, 2012; Transue, Lee and Aldrich, 2009, see also(Barabas and Jerit, 2010)). In one common survey experimental technique, researchers expose participants to an information treatment, often in a question prologue or in a simulated news article, and then assess that exposure’s effect via comparison to a control group. The assumption underlying this design is that differences in the dependent variable identify the main effect of the experimental information. One specific threat to this assumption is the possibility that conditions and/or prevailing news in the real world—including scandals, elections, and wars— mediate experimental effects such that at least some findings are actually interaction effects between the treatment and the environment. In this paper, we directly investigate this issue in the substantive context of conspiracy theories. We simultaneously show that environmental conditions moderate survey experiments and that macro factors affect conspiracy attitudes. As we explain below, our investigation of these substantive questions is inseparable from, and takes advantage of, our methodological queries and contributions. While observational research—particularly scholarship focused on media frames—frequently takes potential priming from external events into account (Mutz, 1994; Kinder, 1998; Hopkins, 2010), these issues’ implications for experiments remain understudied and seemingly under-appreciated. Indeed, while the idea that public opinion is fickle is widely accepted in survey research, standard practice in reporting experimental results does not include discussing potential interactions with contemporaneous conditions. A quick look at the survey experiments papers (about 15 in total) published in the APSR, AJPS, and JOP in 2014 suggests that doing so is exceeding rare (see Figure A1 in the Appendix). None of the