Field Experiments in Political Science: A Brief History and Some Illustrative Examples Don Green...
-
Upload
curtis-flowers -
Category
Documents
-
view
215 -
download
0
Transcript of Field Experiments in Political Science: A Brief History and Some Illustrative Examples Don Green...
Field Experiments in Political Science:
A Brief History and Some Illustrative Examples
Don Green
Columbia University
What is an experiment?
Units of analysis are randomly assigned with known probability to treatment and control conditions
Distinction between random and haphazard assignment
Distinction between random assignment and random sampling
In the physical sciences, perfectly controlled experiments substitute for randomized trials
What is a field experiment?
Random assignment takes place in a naturalistic setting, enhancing generalizability
Ideally, experimenters play an unobtrusive role, reducing the risk of a violation of symmetry between treatment and control
Four dimensions of naturalism: subjects, treatments, contexts, outcome measures
Early (near random?) field experiments in Political Science
Harold Gosnell: Mobilizing voters in the 1924 and 1925 elections
George Hartmann: Using rational or emotional appeals to increase the socialist vote in Allentown, PA in the 1935 elections
Underhill Moore and Charles Callahan: examined the effects of varying New Haven’s parking regulations, traffic controls, and police enforcement in 1930s
Early Field Experiments using Random Assignment
Hovland, Lumdsdaine, and Sheffield: Propaganda studies conducted for Experimental Section of the Research Division of the War Department
Samuel Eldersveld: Partisan and nonpartisan voter mobilization campaigns in the 1953 and 1954 Ann Arbor, MI elections
Ironic that randomized field experiments should die out just as they are gaining prominence in other disciplines (e.g., polio vaccine trials)
Dominant Modes of Behavioral Research in Political Science Surveys: spearheaded in the 1950s by
American National Election Studies using random sampling
Econometric analysis of aggregate data over time and/or space: growing computing power and technical facility of the discipline during 1970s
Fundamental trade off between research design and post hoc statistical correctives
Illusion of Observational Learning Theorem (Gerber, Green, and Kaplan 2004)
When confronted with mixture of observational and experimental evidence, Bayes’ Rule says assign zero weight to observational evidence unless you have informative priors about its bias
When confronted with laboratory experimental evidence, assign it zero weight unless you have informative priors about the biases associated with your extrapolation to some population/setting of interest
Implications of the “Illusion” for scientific practice
Researchers using observational data are oblivious to the fact that they routinely underreport the true degree of statistical uncertainty associated with their findings
Misallocation of the discipline’s research portfolio: No field experiments conducted between 1985 and 1998
Misconceptions about meta-analysis
Reclaiming the Experimental Tradition in Social Science
Reassessment of evidence for a variety of basic behavioral propositions by subjecting them to the same level of scrutiny as pharmaceutical evaluations
Secondary aim is to stimulate experimental reflection even in domains of political science where experimentation is infeasible
What follows is a brief overview of selected projects, some of which refute the “it can’t be done” critique
Project 1: Voter Mobilization(Authors: Gerber and Green)
1998 New Haven Study, N=31,098 Interventions: nonpartisan face-to-face
canvassing, commercial phone banks, direct mail Voter turnout measured using public records Subsequently replicated with hundreds of
thousands of observations in a variety of sites
Synthesis of recent randomized experiments on voter turnout: door-to-door canvassing, leafleting, phone calls, direct mail, and e-mail
Forest Plot of 85 direct
mail experiments – excluding
“social pressure“Studies
ATE = 0.109 ppts
(-0.07,0.290)
Lessons Learned
Face-to-Face canvassing raises turnout by approximately 7-9 percentage-points
Volunteer phone banks have moderate effects (3-5 percentage-points)
commercial phone banks are typically ineffective, as are robo-calls
Direct mail has weak effects (except as noted below)
E-mail has no apparent effect
Project 2: Habit Formation(Authors: Gerber, Green, and Shachar)
Infer habit from long-term effects of randomized intervention
Example of “downstream experimentation” Follow up study of people in the 1998 New
Haven Study showed that treatment group voted at higher rates
Subsequently replicated in 4 of 5 studies For each 100 additional votes generated in this
election, an additional 33 votes are generated in the next election
Project 3: Interpersonal Influence(Author: Nickerson)
Inference without problems of unobserved heterogeneity, which plague other influence studies
Placebo-control design: 2 voter households receive either get-out-the-vote message or a recycling appeal
Also, a control group gets nothing (as expected, turnout is significantly higher in GOTV vs. control)
Project 3: Interpersonal Influence(Author: Nickerson)
Findings show that housemates of registered voters who were contacted in the treatment group were significantly more likely to vote than housemates of those who were contacted in the placebo group
Example: Nickerson’s influence experiment in Denver 2002
05
101520253035404550
Vot
er T
urno
ut
Voters at the Door Housemates
Recycling (Placebo)GOTV (Treatment)
Project(s) 4: Influence of Television and Radio
Cable system experiments in 2003, 2004: influence of GOTV ads on turnout
Radio ads in 2005 and 2006 municipal elections: influence on competitiveness
Broadcast TV, cable TV, and radio in the context of a $2 million gubernatorial campaign
Influence of radio on ethnic reconciliation in Rwanda
Spanish language radio and voter turnout
Project 5: Social Pressure and Voter Turnout (Gerber, Green, and Larimer)
Using field experiments to test basic behavioral theories
Longstanding interest in “prescriptive” norms dating back to Gosnell’s work in the 1920s
To what extent can one manipulate the salience of “extrinsic” incentives associated with voting?
Study Design: August 2006
Sample: 180,002 households in Michigan, registered voters who voted in 2004
Setting: August primary election, “open” but contested only on the Republican side
Assignment: 10,000 clusters of 18 households each; in each cluster, households assigned at random to one of five groups: Control, Civic Duty, Hawthorne, Self, and Neighbors
Outcome: Voting in the primary election, as indicated by official records for each individual
For more information: (517) 351-1975email: [email protected] Political ConsultingP. O. Box 6249East Lansing, MI 48826
ECRLOT **C050THE WAYNE FAMILY9999 OAK STFLINT MI 48507
Dear Registered Voter:WHO VOTES IS PUBLIC INFORMATION!Why do so many people fail to vote? We've been talking about the problemfor years, but it only seems to get worse.This year, we're taking a different approach. We are reminding peoplethat who votes is a matter of public record.The chart shows your name from the list of registered voters, showingpast votes, as well as an empty box which we will fill in to show whetheryou vote in the August 8 primary election. We intend to mail you anupdated chart when we have that information.We will leave the box blank if you do not vote.DO YOUR CIVIC DUTY - VOTE!-----------------------------------------------------------OAK ST Aug 04 Nov 04 Aug 069999 ROBERT SMITH Voted ______9999 LAURA BETH Voted Voted ______
21
For more information: (517) 351-1975email: [email protected] Political ConsultingP. O. Box 6249East Lansing, MI 48826
ECRLOT **C050THE JACKSON FAMILY9999 MAPLE DRFLINT MI 48507
Dear Registered Voter:WHAT IF YOUR NEIGHBORS KNEW WHETHER YOU VOTED?Why do so many people fail to vote? We've been talking about the problem foryears, but it only seems to get worse. This year, we're taking a new approach.We're sending this mailing to you and your neighbors to publicize who does anddoes not vote.The chart shows the names of some of your neighbors, showing which have voted inthe past. After the August 8 election, we intend to mail an updated chart. Youand your neighbors will all know who voted and who did not.DO YOUR CIVIC DUTY - VOTE!-----------------------------------------------------------MAPLE DR Aug 04 Nov 04 Aug 069995 JOSEPH JAMES SMITH Voted Voted ______9995 JENNIFER KAY SMITH Voted
______9997 RICHARD B JACKSON Voted ______9999 KATHY MARIE JACKSON Voted
______9999 BRIAN JOSEPH JACKSON Voted ______9991 JENNIFER KAY THOMPSON Voted ______9991 BOB R THOMPSON Voted
______9993 BILL S SMITH
______9989 WILLIAM LUKE CASPER Voted ______9989 JENNIFER SUE CASPER Voted
______9987 MARIA S JOHNSON Voted Voted
______9987 TOM JACK JOHNSON Voted Voted
______
22
2006 August Primary
Election
Experimental Group
Control Civic Duty Hawthorne Self Neighbors
Percent Voting
29.7% 31.5% 32.2% 34.5% 37.8%
N of Individuals
191,243 38,218 38,204 38,218 38,201
All contrasts with the control group are significant at p < .001, two-tailed test, using robust cluster standard errors (clustered at the household level).
Project 6: The Effects of Criminal Sentences on Recidivism (Green and Winik)
Random assignment of judges creates analytic leverage
~1000 defendants in Washington, DC drug courts randomly assigned to nine “calendars” with different sentencing proclivities
No effect of incarceration or length of sentence on recidivism
Miscellaneous Randomized Experiments of Interest in the Domain of Political Attiudes and Actions Randomly varying rules about legislative
seniority, term length, and floor recognition Discrimination experiments focusing on the
responsiveness of legislators to constituents of varying ethnic, racial, or partisan profile
Effects of audits and “accountability” interventions
Exposure to draft, school, or visa lotteries and their effects on attitudes and behavior
Bottom Line
Randomized experimentation in real world settings give social scientists access to the kinds of practical knowledge that outsiders care about
Possibility, of course, for premature extrapolation of experimental results: e.g., class size experiments
Importance of creating firm empirical foundation for theoretical development and policy intervention (e.g., prejudice reduction)
For a recent (2012) discussion of social science experiments and experimental design, see…