From experimental to quasi- experimental research · From experimental to quasi-experimental...

34
Research Methods for Political Science Dr. Thomas Chadefaux Assistant Professor in Political Science [email protected] 1 From experimental to quasi- experimental research

Transcript of From experimental to quasi- experimental research · From experimental to quasi-experimental...

Page 1: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Research Methods for Political Science

Dr. Thomas ChadefauxAssistant Professor in Political Science

[email protected]

From experimental to quasi-experimental research

Page 2: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Experiments…

… Address threats to internal validity:– Omitted variables– Trends in outcomes (e.g., inflation, aging, etc.)– Mis-specified variances (e.g., autocorrelation or

group correlation)– Mismeasurement– Endogeneity– Selection effects.• E.g., effect of training programs on earning

2

Page 3: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Experiments…

• … are less able to address threats to externalvalidity:– Unrepresentative treatment group of the

population– The effect of the treatment might differ

depending on the setting (lab vs real life)– The effect of the treatment may differ over time

3

Page 4: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Experiments and quasi-experiments

• Experiments are the gold standard because of random assignment

• But in the real-world, random assignment is often not possible.

4

Page 5: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Quasi-experiments

• Quasi-experiments try to be as close as possible to experiments, but without the experimenter’s ability to assign subjects to one group or another

• -> take advantage of situations occurring naturally that correct for issues of internal and external validity

5

Page 6: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Main types of quasi-experiments

• Difference in differences

• Natural experiments• Instrumental variables• Regression-discontinuity designs

6

Page 7: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

DIFFERENCE IN DIFFERENCES

7

Page 8: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Difference in differences

8

Page 9: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

An example

9

New Jersey

Pennsylvania

Empl

oym

ent

February NovemberApril

Page 10: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Disadvantage of DiD

• Parallel trend assumption

10

Page 11: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Outcome

TimeYear0Yeart-1Yeart-2Yeart-3Yeart-4

Treatment

Comparison

Start of intervention

Matched Comparison

Matchedcounterfactual

Even better: matched counterfactuals: find a good match, or a synthetic comparison

Page 12: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Another example: Effect of anti-speeding measuresin Connecticut on fatalities

before and after measures

12

Source for this and upcoming images: Campbell, D. T. (1968). The Connecticut Crackdown on Speeding: Time-Series Data in Quasi-Experimental Analysis. Law and Society Review, 3(1), 33–54.

Page 13: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Time series2. interrupted time series

13

Page 14: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Time series3. Multiple time series

14

Page 15: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

15

Time series3. Multiple time series

Page 16: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

NATURAL EXPERIMENTS

16

Page 17: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Natural Experiments

• A classic example:1854 cholera outbreak in London

17

Page 18: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Types of natural experiments• Lottery studies

– Vietnam draft– Lottery winners and socio-economic outcomes.

• Among lottery winners, winnings are randomly assigned– Public policies

• In Bogotá, vouchers that covered the cost of private secondary school were allocated by lottery– Evidence of persistent effects on academic achievement

– Term lengths in legislatures. Does length improve performance (e.g., abstain less, propose more bills?)• Argentine, 1983. To introduce a staggered system, some legislators were

assigned 2-year initial terms– Effect of indiscriminate violence on insurgent attacks -> drunk Russian

soldiers– soldiers “get drunk as pigs, lob out a few shells, claim combat pay and

get drunk again”

18

Page 19: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Jurisdictional borders

• Borders that separate similar populations, firms, etc.

• An intervention affects one side, but not the other

• E.g., Posner on Chewas and Tumbukas

19

Page 20: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

INSTRUMENTAL VARIABLES

20

Page 21: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Instrumental variables

• Same idea as standard natural experiments: find an additional variable (an ‘instrument’) that is correlated with the independent variable but could not be influenced by the DV or correlated with its other causes

21

Page 22: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Instrumental variablesExample 1

22

Size of Tea Party Rally

How conservativedoes represen-tative vote in thelegislature?

Madestam, Andreas, D Shoag, S. Veuger, and D. Yanagizawa-Drott. 2013. “Do Political Protests Matter? Evidence from the Tea Party Movement.” The Quarterly Journal of Economics 128(4): 1633–85.

Page 23: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Instrumental variables

23

Size of Tea Party Rally

How conservativedoes represen-tative vote in thelegislature?

Madestam, Andreas, D Shoag, S. Veuger, and D. Yanagizawa-Drott. 2013. “Do Political Protests Matter? Evidence from the Tea Party Movement.” The Quarterly Journal of Economics 128(4): 1633–85.

Politicalpreferences

Page 24: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Instrumental variables

24

Size of Tea Party Rally

How conservativedoes represen-tative vote in thelegislature?

Rainfall on the day of the rally (conditional on chance of rain)

Rainfall cannot be the consequence of political beliefs, culture of activism or previous voting behaviour. But it is likely that it affects turnout at rallies.

Politicalpreferences

Page 25: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Instrumental variablesExample 2

• An example: effect of resource scarcity on conflict? Problem of reverse causality.

• Solution: use as-if random assignment to scarcity using rainfall patterns.

25

Page 26: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Instrumental variablesExample 2

26

Resource scarcity conflict

Page 27: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Instrumental variablesExample 2

27

Resource scarcity Conflict

Rainfall

Page 28: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Instrumental variablesExample 3

• Vietnam war draft– Angrist, Joshua D. (1990). "Lifetime Earnings and the Vietnam Draft Lottery:

Evidence from Social Security Administrative Records". American Economic Review. 80 (3): 313–336

29

Page 29: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Example 3

30

Soldier Earnings

Personality

Page 30: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Example 3

31

Soldier Earnings

Personality

Draft (lottery)

Page 31: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

REGRESSION DISCONTINUITY DESIGNS

32

Page 32: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Discontinuity design

33

Page 33: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

34

Page 34: From experimental to quasi- experimental research · From experimental to quasi-experimental research. ... one group or another •-> take advantage of situations occurring naturally

Regression-discontinuity designs

35The Impact of Secondary Schooling in Kenya: A Regression Discontinuity Analysis. Owen Ozier, 2011