iatgen: A Free, User-Friendly Package forImplicit Association Tests (IATs) in Qualtrics

BackgroundThe Implicit Association Test (IAT; Greenwald et al., 1998) allows researchers to assess the strength of association between pairs of targets (e.g., Black vs. White) and attributes (e.g., Good vs. Bad). The IAT has seen widespread use, for example, to study brand preference, social attitudes, morality, self-esteem, and many other social-cognitive constructs (e.g., Greenwald, Poehlman, Uhlmann, & Banaji, 2009). Although web-based research is gaining popularity (e.g., Qualtrics, MTurk, etc.; Paolacci & Chandler, 2014), there is no easy way to run IATs in web surveys. Commercial IAT software is expensive, cumbersome, and does not integrate well with mainstream survey software. In response, we created iatgen, a free, easy-to-use tool for building and analyzing IATs via common survey software. It provides full integration with Qualtrics, automates data cleaning and export, and is ideal for lab and online use.

iatgenIatgen consists of IAT building and analysis tools in both R package and web applet format (https://applibs.shinyapps.io/iatui/).

How iatgen works: iatgen creates HTML and JavaScript files that are added into a special Qualtrics template to make a functional IAT. All stimuli are pre-loaded, so internet speed does not influence results. By default, iatgen counterbalances left/right starting configurations of targets and attributes and uses a 40-trial washout block to avoid any order effects within the IAT (Nosek, Greenwald, & Banaji, 2005).

Building IATs: • In Shiny: The Shiny web applet automates IAT production (see Figure 2). Users configure the IAT via

the applet (image URLs can be included, but images must be hosted by the user, preferably in a Qualtrics account). Iatgen then provides a ready-to-run Qualtrics survey for download, import, and customization. The survey can then be customized like any other survey (e.g., allowing full use of survey flow).

• In R: The R interface offers more advanced features (e.g., custom number of trials per block and multi-IAT designs). Users simply run a script which generates HTML and JavaScript files; these are then manually copy/pasted into one of several included templates. Although more work-intensive, this provides greater flexibility in design and includes all the design advantages of the Shiny applet.

Analyzing IATs: Iatgen processes the data via data-cleaning scripts, providing diagnostics such as reliability, error rates, drop counts, data reduction and D-scoring (Greenwald et al. 2003), and data export. This can be done automatically via the web applet or manually in R. The R package offers greater flexibility (e.g., full analysis of all IAT parameters; tools for scoring odd/even trials for CFA; Asendorpf, Banse, & Mücke, 2002). R users can continue in R for data analyses.

A website will soon provide full tutorials, scripts, build and analysis vignettes, video walkthroughs, and files for free at https://iatgen.wordpress.com/ (coming Fall 2016).

Reliability / Validity SamplesData from studies using iatgen are included to demonstrate psychometrics. By default, iatgen cleans all data according to Greenwald et al. (2003) guidelines.

• Study 1: Standard insect–flower evaluation (pleasant vs. unpleasant; Greenwald et al., 1998, Study 1). Participants recruited via Amazon Mechanical Turk (MTurk). Study to be included in iatgen manuscript (Carpenter, Pullig, Pogacar, LaBouff, Kouril, Isenbreg, & Chakroff, in preparation).

• Study 2: Implicit distrust of atheists (atheist vs theist; trust vs. distrust). Recruited via subject pool at public university in the Northeastern US. To be included in iatgen manuscript (Carpenter et al., in prep).

• Study 3: Two IATs were given: pleasure associations (pleasure vs. neutral) and preferences (good vs. bad). Participants were randomly assigned to complete each IAT for one one of the following pairs of targets: competing mouthwashes (Scope vs. Listerine), batteries (Duracell vs. Panasonic), or ice creams (Haagen Dazs vs. Ben & Jerry). Recruited via MTurk. Data previously reported by Carpenter and Pullig(2015).

• Study 4: Same IATs given for mouthwashes. Recruited via MTurk. Data previously reported by Carpenter and Pullig (2015).

• Study 5: Both IATs from Studies 2-3 and a healthy-unhealthy IAT for ice cream vs. vegetables. Participants exposed to healthy-pleasure or unhealthy-pleasure prime prior to the IATs. Recruited via MTurk. Data previously reported by Carpenter and Pullig (2015).

Thomas Carpenter1,2, Chris P. Pullig2,Ruth Pogacar3, Jordan LaBouff4,

Michal Kouril5, Naomi Isenberg1, & Alek Chakroff6

1 Seattle Pacific University, Seattle, WA2 Baylor University, Waco, TX

3 University of Cincinnati, Cincinnati, OH4 University of Maine, Orono, ME

5 Cincinnati Children’s Hospital Medical Center, Cincinnati, OH6 Harvard University, Cambridge, MA

Results• Data were analyzed for 3,629 separate IATs, generated using iatgen and run in Qualtrics.

• Key psychometrics are shown in Table 1. IATs exhibited acceptable psychometrics:

• Reliability was acceptable, n-weighted average = .78.

• Drop rates (dropped if critical block(s) > 10% of responses < 350 ms) were low at 2.3%.

• Error rates (% of trials answered incorrectly) were low; n-weighted average = 5.1%.

• Timeout rates (% of trials > 10,000 ms latency) were low, n-weighted average = 0.03%.

• Most IATs correlated in theoretically predicted directions with explicit measures at predicted levels.

• Sample 3 was used to test iatgen’s multi-IAT tools and included two IATs (pleasure and preference) across 3 product replicates. Iatgen scored the IAT separately for odd/even trials by product; CFAs were analyzed using lavaan for R. The two-factor model was supported across products (similar results were obtained Samples 4-5, not shown).

• Mouthwash: Dc2(1) = 76.50, p < .001, DAIC = -74.50, DBIC = -71.23.

• Batteries: Dc2(1) = 61.89, p < .001, DAIC = -59.89, DBIC = -56.73.

• Ice cream: Dc2(1) = 89.32, p < .001, DAIC = -87.32, DBIC = -84.02.

Conclusions§ IATs built using iatgen run successful in Qualtrics and exhibit appropriate psychometrics and

correlations for explicit measures comparable with results of Hofmann et al. (2005). Implicit/explicit correspondence was also comparable to that reported in existing research (Hofmann et al., 2005).

§ Iatgen offers advantages over other IAT tools: it is free, flexible, and can be fully integrated into Qualtrics. For example, it was possible to randomize IAT content in Study 3 and to counterbalance the order of IATs / explicit measures in multi-IAT samples.

§ The ability to deploy multiple IATs and conduct CFA analysis on IATs is desirable, given the proliferation of IATs in the literature.

§ Note: iatgen is still in development. Interested users should contact the first author at tcarpenter@spu.edu prior to use.

ReferencesAsendorpf, J. B., Banse, R., & Mücke, D. (2002). Double dissociation between implicit and explicit personality self-concept: The case of shy behavior. Journal of Personality and Social

Psychology, 83, 380–393. Carpenter, T. P., & Pullig, C. P. (2015, August). Don’t tempt me: A dual-process examination of implicit pleasure associations and preferences. Paper session presented at the Annual

meeting for the American Psychological Association and the Society for Consumer Psychology, Toronto.Carpenter, T. P., Pullig., C. P., Pogacar, R., LaBouff, J., Kouril, M., Isenberg, N., & Chakroff, A. (in preparation). Measuring implicit cognition in Qualtrics with iatgen: A free, user-friendly tool

for building survey-based IATs. Manuscript in preparation.Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social

Psychology, 74, 1464–1480. Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003). Understanding and using the Implicit Association Test: I. An improved scoring algorithm. Journal of Personality and Social Psychology,

85, 197–216. Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009). Understanding and using the Implicit Association Test: III. Meta analysis of predictive validity. Journal of Personality

and Social Psychology, 97, 17–41. Hofmann, W., Gawronski, B., Gschwendner, T., Le, H., & Schmitt, M. (2005). A meta-analysis on the correlation between the Implicit Association Test and explicit self-report measures.

Personality and Social Psychology Bulletin, 31, 1369–1385.Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2005). Understanding and using the implicit association test: II. Method variables and construct validity. Personality and Social Psychology

Bulletin, 31, 166–180Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184–188.

Figure 1. Left: an insect-flower IAT running in Qualtrics built using iatgen. Center: an same IAT being generated within the web applet. Right: the same IAT generated via an R build script.

Note: + p < .06, * p < 0.05, ** p < 0.01, *** p < 0.001. Rel = reliability estimate (split-half based on odd/even trials with spearman-brown correction). Explicit measures by sample: Sample 1 = pro-flower attitudes (r1) and anti-insect attitudes (r2); Sample 2 = pro-atheist attitudes (r1) and anti-atheist attitudes (r2); Sample 3-5 = product preference and pleasure scales (correlated as appropriate); Sample 5 = additional relative-health items. Sample 5 health IAT stimuli focused on fat/thin, whereas sample 5 health items assessed food healthiness.

Table 1. Psychometric Properties of Qualtrics IATs Generated in Iatgen in 5 Samples.

Figure 2. An image-based IAT running in Qualtrics.