RUNNING HEAD: PREDICTIVE VALIDITY OF THE IAT Understanding and Using the Implicit Association Test: III. Meta-analysis of Predictive Validity

129 words) This review of 103 studies (140 independent samples, 10,967 subjects), found average r=.27 for prediction of a wide collection of behavioral, judgment, and physiological measures by Implicit Association Test (IAT) measures. Parallel explicit (i.e., self-report) measures, available in 2/3 of these studies, also predicted effectively (96 samples, 7,018 subjects, average r=.33), but were much more heterogeneous in magnitude. Predictive validity of self-report (but not IAT) was impaired in socially sensitive domains, presumably due to impression-management-related distortions of self-report. For studies of intergroup discrimination (1/3 of all studies) IAT measures significantly exceeded self-report measures in predictive validity. The more highly IAT and self-report measures were correlated, the greater the predictive validity of both. Both IAT and self-report measures possessed incremental validity, predicting criterion variance beyond that explained by the other. Greenwald et al. Predictive validity of the IAT (Draft of 12 Oct 2007) 3 In press, JPSP. This is NOT a final draft; numerical results will be updated prior to publication Understanding and Using the Implicit Association Test: III. Meta-analysis of Predictive Validity In the first handbook-chapter review of the concept of attitude, Gordon Allport (1935) characterized attitude as social psychology’s “most distinctive and indispensable concept.” That characterization has been accepted by scholars ever since, even when the attitude construct was enmeshed in a crisis of predictive validity triggered by Alan Wicker’s (1969) review. Wicker found remarkably little evidence to support the idea that attitudes predicted behavior toward the attitudes’ objects (cf. Festinger, 1964). Social psychologists were, during the 1970s, obliged to contemplate the possibility that their attitude construct might not deserve the lofty position accorded it by Allport. In fairness to the attitude construct, there had not been many studies of its predictive validity prior to Wicker’s 1969 review. In the first meta-analysis of attitude–behavior relation studies, Kraus (1995) included only 13 pre-1969 publications. When, in response to Wicker’s review, social psychologists began to address this empirical deficit, they found it initially difficult to produce strong evidence for the predictive validity of attitudes. Nevertheless, by the early 1980s several researchers, especially Ajzen and Fishbein (e.g., 1977) and Fazio and Zanna (e.g., 1981), had established the predictive validity of attitude measures, effectively restoring the attitude construct to its prior status (see also Kelman, 1974). By 1995, sixty years after Allport had hailed attitude as social psychology’s central construct, Kraus was able to assemble results from 88 studies to estimate an average predictive validity effect size of r = .38. Research of the 1970s and 1980s on attitude–behavior relations established two methods that could usually be relied on to produce at least moderate effect sizes for attitude–behavior correlations. The first was a refinement of self-report methods for measuring attitudes, to ensure Greenwald et al. Predictive validity of the IAT (Draft of 12 Oct 2007) 4 In press, JPSP. This is NOT a final draft; numerical results will be updated prior to publication that attitude measures were worded in ways that corresponded closely to the measures of behavior with which their correlations were being examined (Ajzen & Fishbein, 1977). The second was to identify and capitalize on moderator variables that influenced the strength of attitude–behavior correlations, such as the personal importance of the attitude and its stability across time (Krosnick, 1988). Recent findings that have revealed attitudinal processes for which their possessors may have limited awareness suggest the additional possibility of drawing on implicit measurement techniques to reveal aspects of attitudes that may not be within the reach of self-report measures (e.g., Bargh, Chaiken, Govender & Pratto, 1992; Bargh & Chartrand, 1999; Dovidio, Kawakami, Johnson, Johnson, & Howard, 1997; Fazio, Jackson, Dunton, & Williams, 1995; Greenwald & Banaji, 1995; Hetts, Sakuma, & Pelham, 1999; Jones, Pelham, Mirenberg, & Hetts, 2002; Nisbett & Wilson, 1977; von Hippel, Sekaquaptewa, & Vargas, 1997; Wittenbrink, Judd & Park, 1997). The task of determining what implicit measures of attitudes can predict has been pursued most extensively with one particular method for measuring implicit social–cognitive constructs, the Implicit Association Test (Greenwald, McGhee, & Schwartz, 1998; Nosek, Greenwald, & Banaji, 2007). This article summarizes findings of research that has evaluated the predictive validity of Implicit Association Test measures. The Implicit Association Test (IAT) IAT measures assess strengths of associations between concepts by observing response latencies in computer-administered categorization tasks. In an initial block of trials, exemplars of two contrasted concepts (e.g., racially Black and White face images) appear on a screen and subjects rapidly classify them by pressing one of two keys (for example, “d” for Black and “k” for White). Next, exemplars of another pair of contrasted concepts (for example, words Greenwald et al. Predictive validity of the IAT (Draft of 12 Oct 2007) 5 In press, JPSP. This is NOT a final draft; numerical results will be updated prior to publication representative of positive and negative valence) are also classified, using the same two keys. In a first combined task, exemplars of all four categories are classified, with each assigned to the same key as in the initial two blocks (e.g., “d” for White or positive and “k” for Black or negative). In a second combined task, a complementary pairing is used (i.e., “d” for White or negative and “k” for Black or positive). Differences in average latency between the two combined tasks provide the basis for the IAT measure. For example, faster responses for the {White+positive|Black+negative} combined task than for {White+negative|Black+positive} indicate a stronger association of White than of Black with positive valence. Research conducted since the initial 1998 publication of the IAT has provided substantial evidence concerning psychometric properties of IAT measures (Egloff & Schmukle, 2002; Greenwald & Nosek, 2001; Greenwald & Farnham, 2000; Lane, Banaji, Nosek, & Greenwald, 2007; Nosek et al., 2007; Rudman, Greenwald, Mellott, & Schwartz, 1999). It has been found that IAT measures have satisfactory internal consistency (Bosson, Swann, & Pennebaker, 2000; Dasgupta & Greenwald, 2001; Greenwald & Nosek, 2001; Greenwald & Farnham, 2001), within broad limits are not determined by subjects’ familiarity with IAT stimuli (Dasgupta, McGhee, Greenwald, & Banaji, 2000; Ottaway, Hayden, & Oakes, 2001; Rudman et al., 1999), and are relatively insensitive to methodological factors such as the number of trials, the number of exemplars per concept, and the time interval between trials (Nosek, Greenwald, & Banaji, 2005; Greenwald et al., 1998). Test-retest reliability of IAT measures was recently reported to have a median value of r = .56, across nine available reports (Nosek et al., 2007). A useful property of IAT measures is their presumed reliance on associative processes that can operate automatically (Devine, Plant, Amodio, Harmon-Jones, & Vance, 2002; Greenwald et al., 2002). (See Conrey, Sherman, Gawronski, Hugenberg, & Groom, 2005, for an Greenwald et al. Predictive validity of the IAT (Draft of 12 Oct 2007) 6 In press, JPSP. This is NOT a final draft; numerical results will be updated prior to publication investigation aimed at distinguishing the contributions of automatic and controlled processes to IAT measures.) The sensitivity of IAT measures to automatically activated associations is sometimes credited with making IAT scores resistant (even if not immune) to faking. Subjects instructed to fake positive attitudes towards gay men were able to do so on a self-report questionnaire but not on a homosexual–heterosexual attitude IAT (Banse, Seise, & Zerbes, 2001). Asendorpf, Banse, and Mücke (2002) obtained similar findings with a shyness selfconcept IAT, as did Kim (2003) with a race attitude IAT measure. Similarly, subjects instructed to make a good impression in a job application scenario easily altered their self-report responses to appear low in anxiety, but their scores on an anxiety self-concept IAT were relatively unaffected (Egloff & Schmukle, 2002). Although subjects who are explicitly instructed to slow their responding in one of the IAT’s two combined tasks can produce faked scores, naïve subjects do not readily discover this strategy (Kim, 2003; Steffens, 2004; but cf. Fiedler & Bluemke, 2005). The widespread acceptance of the IAT as a measure of association strengths has created a situation similar to that which existed for self-report attitude measures at the time of Wicker’s (1969) review. There is a need to appraise the IAT’s ability to predict relevant social behavior (Banaji, 2001; Fazio & Olson, 2003; Karpinski & Hilton, 2001; Olson & Fazio, 2004). Importance of appraising the predictive validity of IAT measures is further prompted by expressions of interest in using IAT measures for applications in law, policy, and business (Ayres, 2001; Banaji & Bhaskar, 2000; Banaji & Dasgupta, 1998; Chugh, 2004). Additionally, evaluation of the predictive validity of the IAT can play an important role in appraising the construct validity of IAT and other implicit attitude measures (Arkes & Tetlock, 2004; Blanton & Jaccard, 2006; Karpinski & Hilton, 2001; Rothermund & Wentura, 2004). Greenwald et al. Predictive validity of the IAT (Draft of 12 Oct 2007) 7 In press, JPSP. This is NOT a final draft; numerical results will be updated prior to publication In recent investigations, IAT measures have been observed to correlate with many measures of interest, such as anxious behaviors (Asendorpf et al., 2002), partner race preference on an intellectual task (Ashburn-Nardo, Knowles, & Monteith, 2003), math SAT scores (Nosek, Banaji, & Greenwald, 2002a), and alcohol consumption over the course of a month (Wiers, Woerden, Smulders, & de Jong, 2002). In other studies IAT measures have failed to predict measures with which a relation was expected (e.g., Karpinski & Hilton, 2001). The present research assessed the predictive validity of IAT measures quantitatively, while also comparing the predictive validity of IAT measures with that of parallel explicit (self-report) measures, which were available for about two-thirds of the studies located for this review. Method Criteria for Study Inclusion This authors’ aim was to include all available studies that reported predictive validity correlations involving four types of IAT measures — attitudes (concept–valence associations), stereotypes (group–trait associations), self-concepts or identities (self–trait or self–group associations), and self-esteem (self–valence associations). A requirement for inclusion was that the predicted (i.e., criterion) measure was itself neither an implicit measure nor an alternativeformat measure of the same construct being measured by the IAT predictor. Excluded, therefore, were studies focusing on correlations among different IAT measures (e.g., Greenwald et al., 2002) or studies in which the use of IAT was limited to investigating correlations between IAT and parallel self-report measures. Numerous studies of the latter type were recently summarized meta-analytically by Hofmann, Gawronski, Gschwendner, Le, and Schmitt (2005) and this was also the subject of a 57-topic study of attitude IAT and self-report measures by Nosek (2005). Additionally excluded by these criteria were studies in which an IAT measure of self–group Greenwald et al. Predictive validity of the IAT (Draft of 12 Oct 2007) 8 In press, JPSP. This is NOT a final draft; numerical results will be updated prior to publication association or group–valence association was correlated with membership in that group. The criterion measures that remained available for meta-analysis included a wide variety of measures of physical actions, judgments, preferences expressed as choices, and physiological reactions. To illustrate the exclusions and inclusions: A study of correlations between an IAT measure of attitude toward mathematics and parallel self-report measures of attitudes (e.g., Nosek, Banaji, & Greenwald, 2002b) was excluded because the correlation was between an IAT measure and self-report measures of the same construct (i.e., attitude towards mathematics). In contrast, correlations between IAT race attitude measures and nonverbal actions toward persons of that race (e.g., McConnell & Leibold, 2001) were included. Known-groups studies that compared (for example) whether Japanese Americans and Korean Americans differed in an IAT measure of associations of positive or negative valence with the concepts Japanese and Korean (Greenwald et al., 1998, Experiment 2) were excluded because the self-identification (e.g., as Japanese American) was (arguably) conceptually close to being an attitude self-report. However, a study that examined correlations of IAT measures of attitudes toward smoking with selfreported smoking status (Swanson, Rudman, & Greenwald, 2001) was included because the selfidentification (as a smoker or not) provided a measure of relevant behavior. Search Method Studies were sought using three methods: (a) PsycINFO search (using the keywords “IAT”, “implicit association test”, “implicit measure”, “implicit attitudes”, “automatic attitudes” or “implicit social cognition”), (b) internet search (using google.com, keywords: “IAT” or “implicit association test”); and (c) email to the Society of Personality and Social Psychology’s mailing list, requesting any in press or unpublished research using IAT measures. The reference sections of the articles thus obtained were further searched for relevant studies. Authors were Greenwald et al. Predictive validity of the IAT (Draft of 12 Oct 2007) 9 In press, JPSP. This is NOT a final draft; numerical results will be updated prior to publication contacted via email seeking needed or useful statistical results that could not be found in available reports, and also were asked to nominate additional studies that had not yet been reported. One hundred three reports, containing 140 statistically independent samples and data from 10,967 subjects, were identified and located for inclusion as of a cutoff date of February 1, 2007. One hundred six different IAT measures, 100 different explicit measures, and 42 different criterion measures were included in these data sets. The Appendix lists the included studies. Calculation of Effect Sizes Each article meeting criteria for inclusion was separated into statistically independent samples (Lipsey & Wilson, 2001, p. 112), for each of which a mean IAT–criterion measure correlation (ICC) was computed. When available from the sample, mean explicit (i.e., selfreport) measure–criterion measure correlations (ECCs), and mean implicit–explicit correlations (IECs) were also obtained. These means were computed using Fisher’s r-to-Z transformation to average all correlations of the same type in each independent sample. Each such mean Z was associated with an inverse variance weight that was computed as (n – 3) where n is the number of subjects in the independent sample (Hedges & Olkin, 1985, p. 333). The 103 included articles provided 140 statistically independent samples for which a total of 441 ICCs were available. Ninety-six of these samples reported a total of 436 ECCs. Description and Coding of Moderators Variables used as moderators that might explain across-sample variance in effect sizes fell into three categories: conceptual, methodological, and publication. Conceptual moderators were variables suggested either by previous reviews of attitude–behavior relations (e.g., Kraus, 1995) or by findings of the developing literature with IAT measures. Methodological moderators included procedural variations that are frequently found in laboratory studies as well Greenwald et al. Predictive validity of the IAT (Draft of 12 Oct 2007) 10 In press, JPSP. This is NOT a final draft; numerical results will be updated prior to publication as others that are specific to IAT studies. Two publication characteristics were used as moderators, publication year and status of report as published or unpublished. Coding of several of the moderators required judgments based on reading details of the report. For these judged variables, three raters made judgments of each study independently. One of the three raters was blind to results of all studies. The other two were aware of the results of different portions of the studies. For all study characteristics that required such judgments, satisfactory inter-rater reliability was observed (Cronbach’s α ≥ .70), and the three raters’ judgments were averaged for use in analyses. Such reliable ratings of study characteristics have been used successfully in previous meta-analyses (e.g., Eagly, Johannesen-Schmidt, & van Engen, 2003). For methodological and other predictors, the few disagreements that occurred among the three judges were resolved by discussion. Conceptual Moderators Social sensitivity. Subjects’ eagerness to be perceived positively is widely assumed to be a potential source of distortion of self-report measures (e.g., Crosby, Bromley, & Saxe, 1980; Crowne & Marlowe, 1960; Dovidio, Kawakami, Johnson, Johnson, & Howard, 1997; Fazio, Jackson, Dunton, & Williams, 1995; Nosek & Banaji, 2002). Consequently, self-report measures in socially sensitive domains — such as self-reported attitudes and beliefs about racial or ethnic groups — might suffer impression-management influences that could reduce their predictive validity. If, as is also widely assumed, IAT measures resist impression management strategies, their predictive validity may be less prone to influence by social sensitivity of the study topic (cf. Asendorpf et al., 2002; Banse et al., 2001; Egloff & Schmukle, 2002; Kim,