Confidence Interval Distributions 1

Methodological recommendations strongly emphasize the routine reporting of effect sizes and associated confidence intervals to express the uncertainty around the primary outcomes. Confidence intervals (CI) for unstandardized effects are easy to construct; However, CI’s for standardized measures such as the standardized mean difference (i.e., Cohen’s d), the Pearson product-moment correlation, partial correlation, or the standardized regression coefficient are far more difficult. The present manuscript develops a general approach for generating confidence intervals that places a single distribution – the confidence interval distribution – around an effect size estimate. Confidence interval distributions for standardized effect sizes are conceptually simpler than traditional approaches, computationally stable, and easier to present and understand. Computer code permitting users to calculate confidence intervals for several standardized effect sizes is included. Confidence interval distributions also provide a clear link between alternative conceptions such as Fisher’s fiducial intervals and Bayesian credible intervals. Confidence Interval Distributions 3 Constructing confidence intervals for standardized effect sizes. Following the recommendations of Wilkinson and the APA Task Force on Statistical Inference (1999), researchers have been encouraged to supplement the traditional null hypothesis test p-values with effect size estimates and corresponding confidence intervals. Providing and examining effect sizes and confidence intervals helps shift the research question from solely asking “Is the effect different from zero?” to inquiring as well “What is the estimated magnitude of the effect and the precision of that estimate?” (see Ozer, 2007, for a discussion of interpreting effect sizes). For statistics like a mean, creating a confidence interval is simple, straightforward, covered in almost every undergraduate introductory statistics textbook, and easily done by hand. However, for standardized effect size estimates – including the standardized mean difference, correlation coefficient, and the standardized regression coefficient among others – creating a confidence interval is complicated, conceptually convoluted, not presented in undergraduate (and many graduate) textbooks, and cannot be done by hand. The conceptual and analytical difficulties in creating these confidence intervals are powerful barriers to the teaching, understanding, and ultimately the adoption of reporting confidence intervals for standardized effect sizes. The purpose of this brief didactic manuscript is to present a more transparent, robust, and simpler methodology for the creation of such confidence intervals. This manuscript first introduces a short empirical example and briefly reviews confidence intervals for mean differences. Current approaches for generating confidence intervals for standardized effect size estimates are then outlined. Finally, how these same intervals can be generated through randomly constructed distributions that provide a visual parallel to traditional confidence Confidence Interval Distributions 4 intervals is illustrated. The present manuscript develops confidence interval distributions for effect sizes (standardized and unstandardized) – distributions whose quantiles provide the correct confidence interval limits. This approach allows the determination of a confidence interval based on quantiles of the same distribution provides a much cleaner and parsimonious account for the generation of an interval and links the approach for obtaining unstandardized and standardized confidence intervals. Illustrative Empirical Example As an illustrative example, the data from Study 2A from Dunn, Biesanz, Finn, and Human (2007) are re-examined. In brief, participants (n=33) were randomly assigned to interact either with a stranger (n 1 = 16 ) or their romantic partner (n 2 = 17 ) during a surreptitiously recorded interaction. Afterwards participants reported their levels of positive affect after the interaction on a 33-point scale. Raters coded how hard participants were trying to self-present during the interaction on a 1-5 point scale. The hypotheses were (a) interacting with a stranger would lead to enhanced self-presentation relative to the romantic partner and (b) in turn, greater selfpresentation would lead to enhanced levels of positive affect. Indeed, interacting with a stranger led to greater rated self-presentation (M1 = 3.21, SD = .55) than with a romantic partner (M2 = 2.20, SD = .57), t(31)=5.16, p<.0001, d=1.80, CI.95 .97, 2.60 !" #$ . In turn, self-presentation, controlling for condition, was strongly associated with greater positive affect, b = 5.42, b * =.76, t(30)=3.67, p<.001, df = n – p – 1 = 33 – 2 – 1, where p is the number of predictors. Throughout the manuscript we denote observed standardized regression coefficients as b* to avoid confusion with the population regression coefficient. Thus b and b* refer to the sample unstandardized and standardized regression coefficients that are estimates of population quantities ! and !,