Experimental Design

Making your experiment more valid and more efficient

So, the basic experimental procedure in social science is:

• Gathering of a subject pool and then random assignment to conditions– Note: the sample is not random, but once you

have a subject pool you flip a coin, etc. to determine which subjects receive which condition

• Depending upon the experimental design, a measure of the dependent variable may be taken from the groups prior to the manipulation

• Subjects are exposed to the ‘treatment’ assigned to their group– May be some level of the factor or a zero-level (control)

treatment– For example, the ‘’experimental group” may play a video

game with lots of violence while the control group plays a non-violent game

• Both groups are then measured on the dependent variable of interest– In this case, some measure of aggressiveness

• The data are then analyzed to test the research hypothesis

Basic design(After-only, control group design)

R X O

R O

R=randomization, X=manipulation, O=observation

What’s the effect?

• The difference between group scores on the dependent variable is assumed to be due to the manipulation

• But what if the groups were different at the start?– Randomization should have taken care of that– However, pure chance can sometimes lead to

differences at the outset• So, measure them before the manipulation as well as

after

Before-after, control group design

R XO1 O2

R O2O1

R=randomization, X=manipulation, O=observation

Before-after, control group design

• Now there’s a new problem:– Measuring the subjects and then measuring them

again later may cause them to respond differently to the second measure than they otherwise would have (true for both experimental and control subjects)

– Measuring the experimental subjects before exposing them to the treatment may change their response to the manipulation

Problems with the basic design

• The difference between control and experimental conditions includes more than violence– Experimenters may have control group play a

nonviolent video game• Many other variables may affect the outcome– For example, gender• Measure and include in statistical analyses• Use factorial design

So:

• There will always be some problems with any procedure

• The goal is to minimize the threats to validity as best your circumstances (including budget) allow– Especially internal validity

How do we do that?

• Experimental design• Statistical analyses• Careful application of procedures

Experimental design

• Two facets– Choices in identifying the independent and

dependent variables to be measured and/or manipulated

– Choices in developing the procedures for manipulation, measurement, environment of testing, etc.

Choosing the variables

• Theory determines variables manipulated and measured

• Experimental designs where the research manipulates more than one independent variable are known as factorial designs– Mediating concepts often are included

Choosing the measures

• The measures are chosen to best represent the concepts in the original theory– Measurement validity

• Must work in an experimental setting– Lengthy questionnaires unlikely to work– Physical measures more easily applied in a

laboratory setting

Factorial designs

• Factorial designs are useful because they:– Are efficient

• With two independent variables we can test three hypotheses instead of just one– Don’t need three times as many subjects

• However, there is a practical limit to independent variables– As the number rises, the required number of subjects increases quickly (5

per cell)

– Help us avoid coming to invalid conclusions• They allow us to control for potentially confounding variables and

interactions– We can test the impact of third variables

• Factorial designs can be more difficult to administer

An example

• We may want to test the following hypothesis:– “Exposure to Dancing with the Stars will lead to

greater liking for the contestants”• We might, however, expect that men and

women will be affected differently by the show

• So, we might add gender as another independent variable in the design

Factorial design

Women

See Dancing with the Stars

Don’t see Dancing with the Stars

All groups are measured on liking for contestants

Men

What we can test

• “Main effect” of exposure to Dancing with the Stars

• “Main effect” of gender• “Interaction effect” of exposure by gender• Remember: the dependent variable is liking

for contestants

Results(% who like contestants)

Didn't see DWTS Saw DWTS0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

MenWomen

Conclusions

• Main effect of gender– Yes: Gender affects liking for celebs

• Main effect of exposure to DWTS– Yes: Exposure increases liking

• Interaction effect of exposure by gender:– Yes: Exposure has a clear positive effect for

women, little or even negative effect for men

Procedural choices

• How many times will you measure the subjects?– You can account for chance fluctuation by

increasing the number of times the groups are measured• You increase the likelihood of testing reactivity, though

R O1 O2X O3 O4

R O1 O2 O3 O4

Additional design concerns

• How many times/how long will the manipulation be presented?– Testing a single exposure to DWTS may not be

valid• Viewers may watch many shows during a season

R O1 X1 O2 X2 O3

R O1 O2 O3

Removal of the treatment

R O1X O2 -X O3

R O1 O2 O3

• How many levels of the independent variable will be presented?– May want to test high v. medium v. low levels of

violence, etc.

R O X(high) O

R O X(low) O

R O X(medium) O

R O O

Factorial nomenclature

• The number of treatments on each factor are crossed– 2X3 factorial design has 2 treatment levels on one

factor and 3 treatment levels on another factor– If we tested the effects of soft, medium and loud

music and gender upon ease in a social situation, we would have a 2X3 factorial study• Factor 1: music volume 3 levels: soft, medium and loud• Factor 2: gender 2 levels: male, female• Dependent variable: ease in a social situation

Additional design concerns

• Artificiality of the testing environment– Labs allow for the greatest control over third

variables• Volume, distance from screen, prevent other people from

walking by, etc.

– Lab experience can sometimes be made less artificial• Some commercial-testing services provide the subjects

with a simulated living room for exposure• Loss of control over third variables weighed against

increase in generalizability (external validity)

Statistical analyses

• If you measure potential confounds you can use statistical procedures to remove their impact on the dependent variable

Careful application of the design

• Unplanned variations among treatments can easily occur when conducting experiments in the real world– The person you wanted to greet everyone and

provide instructions may have to leave for an emergency

– Subjects may talk to each other outside the lab before they have all completed their part of the experiment

– And so on

Remember:

• Experiments are the most effective quantitative research method for testing hypothesized causal relationships

• Experiments emphasize internal validity but are often weak with regard to external validity

• Many different experimental designs exist, each having a set of advantages and disadvantages

• Your goal is to provide the most valid test of theory your circumstances allow you