Quasi-Experimental Designs - · PDF fileQuasi-Experimental Designs In a quasi-experimental...
Transcript of Quasi-Experimental Designs - · PDF fileQuasi-Experimental Designs In a quasi-experimental...
Quasi-Experimental Designs
In a quasi-experimental design, the researcher lacks control over the assignment to conditions and/or does not manipulate the causal variable of interest.
A quasi-independent variable is not manipulated by the researcher but rather is an event that occurred for other reasons.
Examples
Does smoking cause cancer?
Did 9/11 cause an increase in prejudice against people of middle-eastern decent?
Do Republican vs. Democratic presidents affect the economy?
Do extreme events (i.e., winning the lottery or being paralyzed) affect day-to-day happiness?
Does campus crime affect applicants to a university?
Questionable Internal Validity
No control over assignment of participants to the independent variable, so internal validity of quasi-experiments is always questionable.
However, some quasi-experimental designs are more internally valid than others.
The extent to which a quasi-experimental design can eliminate possible threats to internal validity determines its usefulness.
Quasi-Experiments and Internal Validity
To infer that X causes Y
1. X must precede Y in time
2. X <-> Y must be related to each other
3. all other alternative explanations of the results are eliminated through random assignment or experimental control
With quasi-experimental designs, you can’t rule out ALL alternative explanations, but you can try to minimize them.
One-group pretest-posttest design
Example: Sample of violent adolescent women
Treatment: anger management class
Measures: aggressive behavior 1 year pre/post
If aggressive behavior is lower at T2 than T1, can we conclude this decrease is caused by the treatment?
Time 1 Measure
Treatment
Time 2 Measure
O1 X O2
One-group pretest-posttest design
Internal validity problems?
Maturation effect: between observations, participants could have grown out of aggressive behavior
History effect: something else in the school environment caused a decrease—less overcrowding
Testing effect: the act of assessing aggression led to awareness of their own aggression…
Time 1 Measure
Treatment
Time 2 Measure
O1 X O2
One-group pretest-posttest design
For these reasons, simple pretest-posttest design should never be used without a control condition and other efforts to control third variables.
Time 1 Measure
Treatment
Time 2 Measure
O1 X O2
Nonequivalent Control Group Designs
Researcher obtains a groups of participants who are similar to the group that receives the quasi-independent variable.
Example: Do extreme events (winning lottery, being paralyzed) have a long-term effect on day-to-day happiness?
Nonequivalent groups: posttest only design
Measure both groups after one receives the quasi-independent variable.
Event (X): winning lottery, paralyzing accident
Advantage: unlike pre-post design, not subject to testing effect.
Threat to Internal Validity: Selection bias, you can not be sure the groups were the same before the treatment.
How could we try to address this?
Time 1: Event Time 2: Measure
X O
-- O
Time Series Designs
Measure the dependent variable on several occasions before and after the quasi-independent variable occurs.
T1 T2 T3 T4 T5 T6 T7 T8 T9
O1 O2 O3 O4 X O5 O6 O7 O8
8 Observations 4.0 4.3 3.9 4.2 2.9 3.1 3.0 3.3
2 Observations 4.2 2.9
Example: Does a well publicized crime spree (X) around Howard campus impact on number of observed applicants (O1-O8) to the university?
What if we only had the two observations?
Simple interrupted time series design
Advantage: Can rule out maturation.
Possible Threat to Internal Validity: Contemporary history –could be confounded with some other event that occurred at the same time.
Simple interrupted time series design
T1 T2 T3 T4 T5 T6 T7 T8 T9
O1 O2 O3 O4 X O5 O6 O7 O8
T1 2 3 4 5 6 7 8 9 10 11 12 13
O1 O2 O3 O4 X O6 O7 O8 -X O9 O10 O11 O12
Reputation 5 4 6 5 10 10 9 6 7 5 4
Interrupted Time Series with a Reversal
Example: Howard graduates earn 3 metals in one Olympics, and earn 0 metals in the next Olympics. What impact does that have on Howard’s reputation for athletics?
Shows the effect of adding AND removing quasi-independent variable.
This pattern of results rules out maturation and history.
Time Series Designs
Comparative Time Series Design
Examines two or more variables over time in order to understand how changes in one variable are related to changes in another variable
Provides indirect evidence that the change in one variable may be causing the change in the other variable
Time Series Designs
Control Group Interrupted Time Series Design
nonequivalent control group included that does not receive the quasi-independent variable
helps rule out certain history effects
but could still get local history effects
Experimental:O1 O2 O3 O4 X O5 O6 O7 O8
Control :O1 O2 O3 O4 -- O5 O6 O7 O8
Control Group Interrupted Time Series Example: Economic Output of Cities
0
20
40
60
80
100
120
2001 2002 2003 2004 2005 2006 2007 2008 2009
New Orleans
Tampa FL
FEMA went into New Orleans
Longitudinal vs. Cross-sectional Designs
Design Participant #
1990 1995 2000 2005
Longitudinal
P’s 1-25 Age 5 Age 10 Age 15 Age 20
Cross-sectional
P’s 1-25 Age 5
P’s 26-50 Age 10
P’s 51-74 Age 15
P’s 75-100 Age 20
Longitudinal Designs
The quasi-independent variable is time; nothing has occurred from one observation to the next except for the passage of time
O1 O2 O3 O4 O5
Mostly used by developmental psychologists to study age-related changes in how people think, feel, and behave.
Developmental changes in memory, emotions, self-esteem, personality…
Longitudinal Design Example: Marital Satisfaction (Kurdek, 1999)
1 2 3 4 5 6 7 8 9 10
Years of Marriage
Ma
rita
l S
atisfa
ctio
n
Wife
Husband
Note: Years is within-participants. Would you offer to collect these data as a graduate student?
Longitudinal Designs
Three drawbacks:
Difficult to obtain participants who agree to be in a study over a long period of time
Difficult to keep track of participants once they are in the study
Repeatedly testing a sample requires time, effort, and money
Cross-Sectional Designs
Drawback: Age is confounded with generation. Examples:
1. Texting ability by age (10, 20, 30, 40, 50, and 60 year old Ps)
• Younger people are faster texting
• Does texting speed decrease with age?
• Or are there generational effects
with experience with texting?
2. Prejudice as a function of age.
3. Attitudes toward sex in the media as a function of age.
Evaluating Quasi-Experimental Designs
Quasi-experimental designs can show that:
the presumed causal variable preceded the effect in time
cause and effect covary
Quasi-experimental designs do not:
eliminate all other alternative explanations for the results
reason: no random assignment and no experimental control
Increasing Confidence in Quasi-Experimental Results 1. Measure both the effects of the quasi-IV and the
processes assumed to explain the relationship.
2. Include a noncomparable control condition to rule out global history effects.
3…
4…
3. Find a complex pattern of results that cannot be explained by anything except for your quasi-experimental factor.
4. Critical multiplism – employ multiple approaches that may converge to yield conclusions that are as concrete as those obtained in experimental research.
• Longitudinal and cross-sectional
• quasi and full experiments
• different quasi-experiments for the same factor
• replications in different populations...
Increasing Confidence in Quasi-Experimental Results
Design your own Quasi-Experiment
Points to consider
-Pick an event or set of events & one DV variable
-Is it longitudinal, cross-sectional, time-series,
pre-post test, or post-test?
-Is a non-equivalent control group or reversal included?
-What is you hypothesis (prediction)?
-What confounds threaten internal validity?
-Can you improve the design to address the confounds?