Research Methods in Psychology Complex Designs. Experiments that involve two or more independent...

Research Methods in Psychology

Complex Designs

Complex Designs

Experiments that involve• two or more independent variables studies

simultaneously• at least one dependent variable

Simplest complex design• one independent variable with two levels• one dependent variable

Complex Designs, continued

Factorial combination• combine independent variables in an experiment• pair each level of one IV with each level of the other

IV(s)• example:

Closer examination of the Dittmar, Halliwell, and Ive (2006) study: Barbie and young girls’ body image

They also examined the IV of grade in school (kindergarten, 1st, 2nd) as a natural groups variable


Factorial combination of 2 IVs• version of picture book: Barbie, Emme, neutral• grade: kindergarten, 1st, 2nd • factorial combination: 9 conditions• referred to as a “3 x 3”—2 IVs, each with 3 levels

9872nd

6541st

321kindergarten

Grade

NeutralEmmeBarbie

Version of Picture Book


Factorial combination allowed Dittmar et al. (2006) to examine• overall effect of Version of Picture book

Barbie images caused greater body dissatisfaction than Emme and neutral images

• overall effect of Grade body dissatisfaction increased as grade in school

increased

• combined effect of both IVs together results indicated interesting effects for

combinations of grade and exposure to the images


Guidelines for identifying complex designs• at least two IVs• IVs can be independent groups designs

random groups, natural groups, matched groups

• IVs can be repeated measures designs when independent groups and repeated measures

designs are combined, it’s called a mixed design


Main effects• overall effect of an IV in a complex design

effect on DV as if only that IV was studied

Interaction effects• combined effect of IVs considered

simultaneously• An interaction effect occurs when the effect of

an independent variable differs depending on the level of a 2nd independent variable.

Research Example

Kassin, Goldstein, and Savitsky (2003)• pp. 275–283 in text• Research questions

Do interrogators’ expectations about a suspect’s guilt or innocence influence the interrogation tactics they use?

Do interrogators have a confirmation bias in which their initial beliefs about a suspect’s guilt cause them to interrogate more aggressively?

Research Example, continued

Research design• complex design with 2 levels (a 2 x 2 design)

Interrogator Expectation (random groups)• guilty expectation• innocent expectation

Suspect Status (random groups)• actual guilt• actual innocence

• students participated as interrogators or suspects in a laboratory “mock crime”


• Dependent Variables They measured many, we will focus on:

Number of guilt-presumptive questions the interrogator selects for the interview with suspect

Number of persuasive interrogation techniques used during the interview with the suspect

Ratings of the amount of effort the interrogator used to obtain a confession


• factorial combination of 2 x 2 design→ 4 conditions

Interrogators were led to believe the suspect was innocent and the suspect did not commit the crime

Interrogators were led to believe the suspect was guilty and the suspect did not commit the crime

Actual Innocence

Interrogators were led to believe the suspect was innocent and the suspect actually committed the crime

Interrogators were led to believe the suspect was guilty and the suspect actually committed the crime

Actual GuiltSuspect

Status

InnocentGuilty

Interrogator Expectation


• Hypothesis

Based on behavioral confirmation theory, interrogators were predicted to behave toward the suspect in ways that were consistent with their belief of guilt or innocence. In turn, suspects were predicted to respond in ways that support the interrogator’s belief.


Kassin et al.’s (2003) findings• Main effects

A main effect is the effect of one IV, ignoring (or collapsing across) the effect of the other IV

• Two main effects are possible for each DV Interrogator Expectation Suspect Status


• Main effect of Interrogator Expectation compare Guilty Expectation and Innocent Expectation DV: number of guilt-presumptive questions

2.663.70Actual Innocence

Suspect

Status

2.603.62

2.543.54Actual Guilt

Means for Interrogator Expectation

InnocentGuilty



• Means for Interrogator Expectation Guilty: M = 3.62 ← (3.54 + 3.70) ÷ 2 Innocent: M = 2.60 ← (2.54 + 2.66) ÷ 2

• A test of statistical significance revealed that these two means are statistically different. Interrogators who suspected their suspect to be

guilty chose more guilt-presumptive questions (M = 3.62) than interrogators who expected an innocent suspect (M = 2.60)


• Main effect of Suspect Status compare Actual Guilt and Actual Innocence DV: Number of Persuasive Techniques

Means for

Suspect

Status

7.15

11.4210.8811.96Actual Innocence

Suspect

Status6.597.71Actual Guilt

InnocentGuilty



• Means for Suspect Status Actual Guilt: M = 7.15 ← (7.71 + 6.59) ÷ 2 Actual Innocence: M = 11.41 ← (11.96 + 10.88) ÷ 2

• A test of statistical significance revealed that these two means are statistically different Interrogators who interviewed a suspect who was

actually innocent used more persuasive techniques (M = 11.42) than interrogators who interviewed a suspect who was actually guilty (M = 7.15)


Interaction effects• occurs when the effect of one independent

variable differs depending on the level of a second independent variable

• Kassin et al.’s (2003) experiment look at the effect of suspect status (actual guilt,

innocence) at each level of interrogator expectation variable

• An initial approach to examine interaction effects is the subtraction method


• Interaction effect of Interrogator Expectation X Suspect Status DV: Ratings for effort to obtain a confession

5.857.17Actual Innocence

Suspect Status

–0.29–1.53Difference Between Means

5.565.64Actual Guilt

InnocentGuilty



• Because the outcome of the subtraction method yielded very different values

–1.53and – 0.29

an interaction effect between the IVs is likely.

• A test of statistical significance would be needed to confirm this.


Examine the means to understand the interaction effect• When suspects were actually guilty, the effort to

obtain a confession was not affected by whether the interrogator expected the suspect to be guilty (M = 5.64) or innocent (M = 5.56) Not a statistically significant difference

• However, when the suspects were actually innocent, the effort to obtain a confession was greater when the interrogator expected a guilty suspect (M = 7.17) compared to when the interrogator expected an innocent suspect (M = 5.85) A statistically significant difference


• Interrogators differed in their effort to obtain a confession depending on their expectations and whether the suspect was actually guilty or innocent

= an interaction effect between Suspect Status and Interrogator Expectation independent variables

• The effect of one IV differed depending on the level of the 2nd IV this is the definition of an interaction effect


Graphs (“Figures”) can be used to detect interaction effects easily• An interaction effect is likely when lines in the

graph that display the means are not parallel that is, the lines either intersect, converge, or

diverge

• However, a statistical test is always used to determine whether an interaction is statistically significant

Research Example, continuedDV: Ratings of effort to obtain a confession (means)

Interrogator Expectation Guilty Innocent

0

2

4

6

8

10Actual Guilt

Actual Innocence


ANOVA summary table• Statistical significance

p < .05

• Information in summary table is only useful in conjunction with descriptive statistics (e.g., means) for each condition of the experiment.

• Example DV: effort to obtain a confession Statistically significant effects:

• Interaction effect of Interrogator Expectation X Suspect Status• main effect of Interrogator Expectation• main effect of Suspect Status


ANOVA Summary Table for DV: Effort to obtain a confession Source df SS MS F p

eta2Interrogator Expectation 1 34.82 34.82 4.96 .027

.017

Suspect Status 1 58.48 58.48 8.33 .004.029

Interrogator Expect. X Suspect Status 1 27.59 27.59 3.93 .048 .029

Error 294 2063.9 7.02_______________________________________Total 297 2184.77

Analysis of Complex Designs

Steps for Data Analysis• Check the data for errors and outliers• Summarize the results using descriptive

statistics Factorial design tells you who many means you

need to analyze• e.g., a 2 x 2 → 4 conditions (4 means)

Graph the means

Analysis of Complex Designs, continued

• Confirm what the data reveal. The means in an experiment will not all be the

same There will be some variability Key question: Is the variability greater than chance

(error variation)? Variability greater than chance is attributed to the

effect of the independent variable(s) Null hypothesis testing is used to decide whether

the IVs produced an effect on the DVs In complex designs, Analysis of Variance is used


Analysis of Variance (ANOVA)• tells us whether main effects and interaction

effects are statistically significant• when an effect is statistically significant

we say IV caused the effect of DV assuming experiment is internally valid

• results are presented in an ANOVA Summary Table statistics in ANOVA Summary Table are only

useful when descriptive statistics (e.g., mean) are also considered


How are these effects reported?• Main effect of Interrogator Expectation

On average, interrogators who expected a guilty suspect worked harder to obtain a confession than interrogators who expected an innocent suspect (Ms = 6.40 and 5.71, respectively), F(1, 294) = 4.96, p = .027, d = .26

• Main effect of Suspect StatusOn average, interrogators exerted more effort to obtain a

confession when the suspect was innocent (M = 6.51) than when the suspect was guilty (M = 5.60), F(1, 294) = 8.33, p = .004, d = .34


• Interrogator x Suspect Status interaction effect:

The interrogator Expectation x Suspect Status interaction was statistically significant, F(1, 294) = 3.93, p = .048, η2 = .029. Interrogators worked hardest to obtain a confession when they expected a guilty suspect and interviewed a suspect who was actually innocent (M = 7.17). The mean rating for this cell was statistically greater than the other three cells, which did not differ significantly from each other.


• Omnibus ANOVA initial test of main effects and interaction effects

• If interaction effect is statistically significant, conduct follow-up or “post-hoc” tests of statistical

significance, such as simple main effects comparisons of two means


Guidelines for the analysis of a complex design experiment• Step 1: determine whether interaction effects are

statistically significant in a 2-factor experiment, only 1 interaction effect is possible

• Step 2: If interaction effect is statistically significant, identify source of interaction simple main effects and comparisons of two means

• Then examine whether the main effects of each independent variable are statistically significant


Interaction effects• Definition

Effect of one independent variable differs depending on the level of a second independent variable

• Analyze the simple main effects to determine source of interaction


• Simple main effects the effect of one independent variable at one level

of a 2nd IV for example, the effect of the suspect status IV in

the• Expect-Guilty condition or the• Expect-Innocent condition


• Simple main effect of Suspect Status for the Guilty-Expectation condition

• Statistically significant


0

2

4

6

8

10

Actual Guilt

Actual Innocence


• Simple main effect of Suspect Status for the Innocent-Expectation condition

• Not statistically significant


0

2

4

6

8

10

Actual GuiltActual Innocence


Two other simple main effects in Kassin et al.’s experiment• the simple main effect of

Interrogator Expectation for Guilty Suspects

• the simple main effect of Interrogator Expectation for Innocent Suspects

• Which is statistically significant?


0

2

4

6

8

10Actual Guilt

Actual Innocence

Interaction Effects and Theory Testing

Kassin et al. (2003) showed support for behavioral confirmation theory• “Interrogator expectations [triggered] a range

of behavioral confirmation effects, ultimately biasing perceptions of guilt … leading them to exert the most pressure on innocent suspects” (Kassin et al., 2003, p. 199)

Interaction Effects and External Validity

Interaction effect is not statistically significant• generalize findings across conditions of

experiment• example

Kassin et al.’s findings for number of persuasive techniques used by interrogators

Interrogator Expectation x Suspect Status interaction not statistically significant

Interaction Effects and External Validity, continued

• Interrogators used more persuasive tactics when the suspect was actually innocent than when the suspect was actually guilty This was true in the guilty-expectation condition

and

the innocent-expectation condition

• Effect of suspect status generalized across the levels of interrogator expectation

Interaction Effects and External Validity, continued

• The presence of a statistically significant interaction effect sets limits on the external validity of a finding example: Suspect Status x Interrogator

Expectation interaction for number of persuasive tactics

• Not all interrogators who expected a suspect to be guilty exerted a high degree of effort to obtain a confession

• We can’t generalize findings for effort across guilty and innocent suspects, or across interrogator expectation

the number of persuasive tactics depends on• suspect status and interrogator expectations

Interaction Effects and Ceiling/Floor Effects

Floor and ceiling effects• Sometimes an interaction effect can be

statistically significant “by mistake” This occurs when the means for one or more

condition(s) reach, on average, near • the highest possible score (ceiling effect)• the lowest possible score (floor effect)

• When floor or ceiling effects occur, an interaction effect is uninterpretable

Ceiling Effect, example

• interaction effect between Task Difficulty (easy, hard) and Study Hours (10, 15) hours of study had an

effect only in the hard-test condition, not in the easy-test condition

How do we interpret this interaction when we know the highest possible test score is 50?

0

10

20

30

40

50

10 15

Hours of StudyTes

t Sco

re

EasyHard

Ceiling Effect, example

• If we have enough “room” in our DV to assess the effect of the IV, the interaction effect disappears

• This graph shows two main effects: Study Hours and Test Difficulty 0

10

20

30

40

50

60

70

10 15

Hours of Study

Tes

t Sco

re

EasyHard

Interaction Effects and Natural Groups Design

With complex designs,• researchers can test causal inferences for

natural groups variables but wait … isn’t it impossible to make causal

inferences for natural groups variables? natural groups variables are correlational so, how does one make causal inferences?

• Test a theory for why the natural groups differ

Interaction Effects and Natural Groups Design, continued

Steps for making causal inferences about natural groups variables using complex designs• State your theory.

Why do the groups differ? What is the theoretical process?

• Identify a relevant independent variable. This IV should influence the likelihood that the

theorized process will occur

Interaction Effects and Natural Groups Design, continued

• Look for an interaction effect. The natural groups variable and manipulated IV

should produce a statistically significant interaction effect in the predicted direction

This interaction effect allows a causal inference about why individuals differ

Research Methods in Psychology Complex Designs. Experiments that involve two or more independent...

Documents

Transcript of Research Methods in Psychology Complex Designs. Experiments that involve two or more independent...