WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research...
-
Upload
bryce-richardson -
Category
Documents
-
view
217 -
download
1
Transcript of WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research...
![Page 1: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/1.jpg)
WWC Standards for Regression Discontinuity Study Designs
June 2010
Presentation to the IES Research Conference
John Deke ● Jill Constantine
![Page 2: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/2.jpg)
What is the What Works Clearinghouse (WWC)? Initiative of the U.S. Department of Education's
Institute of Education Sciences
Central and trusted source of scientific evidence for what works in education– Develops and implements standards for reviewing and
synthesizing education research– Assesses the rigor of research evidence on the
effectiveness of interventions
2
![Page 3: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/3.jpg)
What Works Clearinghouse
Evidence reports
Practice guides
Help desk
Users of Information
School boardsSuperintendents and AdvisorsState Education DepartmentsFundersIntermediariesConsultantsTechnical Assistance ProvidersPrincipalsTeachersFederal government State governmentNational groupsPublic
EffectivenessInformation
Universities
ResearchOrganizations
Developers
Producers of Information
ResearchThat Meets Standards
Evidence standards
Improved Student Outcomes
Evidence-Based
Interventions and Practices
Topics
Evidence
3
![Page 4: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/4.jpg)
Current WWC Study Designs Standards
Randomized Control Trials (RCTs)– Well executed RCTs can obtain the highest level of causal
validity and meet evidence standards
Quasi-experimental Designs (QEDs)– Some RCTs and matched comparison designs meet
evidence standards with reservations
Regression Discontinuity Designs (RDDs)
Single-Case Study Designs
4
![Page 5: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/5.jpg)
Components of a Research Design Standard
Establishes a level of rigor that informs causality
Operational– Follows a protocol– Replicable and reliable– Transparent
Can be consistently applied to completed studies
5
![Page 6: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/6.jpg)
What is RDD?
Similar to RCT in that treatment and control groups formed by design, not by unobserved self-selection
Different from RCT in that groups are not formed randomly – they are formed purposefully
If groups formed using a cutoff on a continuous "forcing variable," then it is an RDD
6
![Page 7: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/7.jpg)
Process for Developing Standards
Panel convenes
Standards drafted
Study Review Guide drafted
Standards pilot tested
Standards finalized and approved by IES
7
![Page 8: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/8.jpg)
WWC RD Expert Panel
Panelists included Tom Cook, John Deke, Guido Imbens, J.R. Lockwood, Jack Porter, Peter Schochet, and Jeff Smith
The panel met once with follow-up conference calls over the course of several months
WWC staff drafted the RD standards and incorporated feedback from the panelists
The standards document is ready to be used by the WWC, but can be updated in the future
8
![Page 9: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/9.jpg)
Key Considerations
Should the WWC allow RDD studies to join RCTs as studies that “meet standards without reservations”?
How can we distinguish between RDD studies that meet standards with versus without reservations?
9
![Page 10: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/10.jpg)
Overview of RD Standards Screener to identify studies as being RDD Three possible designations:
1. Meets standards without reservations
2. Meets standards with reservations
3. Does not meet standards
Four individual standards:1. Integrity of the forcing (assignment) variable
2. Attrition
3. Continuity of the outcome-forcing variable relationship
4. Functional form and bandwidth
Reporting requirement for standard errors
10
![Page 11: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/11.jpg)
RDD Screener Treatment assignment is based on a “forcing
variable” – units on one side of a cutoff value are in the treatment group, units on the other side are in the comparison group
The forcing variable must be ordinal with at least 4 values above and 4 values below the cutoff value
There must be no factor confounded with the cutoff value of the forcing variable
11
![Page 12: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/12.jpg)
1. Integrity of the Forcing Variable Primary concern is “manipulation”
Two criteria: institutional integrity and statistical integrity
Both criteria must be met to pass this standard without reservations; one criterion must be met to pass with reservations
12
![Page 13: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/13.jpg)
2. Attrition Primary concern is nonresponse bias
Standard is almost the same as the WWC RCT standard on attrition
One difference: overall attrition and differential attrition can either be reported at the cutoff value of the forcing variable or for the full treatment and comparison samples
13
![Page 14: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/14.jpg)
3. Continuity of Outcome-Forcing Variable Relationship Primary concern is phantom impacts from lack
of “smoothness” in the relationship between the outcome and the forcing variable
Two criteria: baseline equivalence on key covariates and no evidence of unexplained discontinuities away from the cutoff value
Both criteria must be met to pass this standard without reservations
14
![Page 15: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/15.jpg)
4. Functional Form and Bandwidth Primary concern is misspecification bias
Five criteria:A. An adjustment must be made for the forcing variable
B. A graphical analysis must be included and should be consistent with bandwidth/functional form choice
C. Statistical evidence must be provided of an appropriate bandwidth or functional form
D. Model should “interact” forcing variable with treatment status or provide evidence that an interaction is not needed
E. All of the above must be done for every combination of outcome, forcing variable, and cutoff value
All criteria must be met to pass standards without reservations; to meet with reservations, (A and D) plus (B or C)
15
![Page 16: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/16.jpg)
Example of Graphical Analysis
16
![Page 17: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/17.jpg)
Meeting Evidence Standards To meet standards without reservations, must
meet each individual standard without reservations
To meet standards with reservations, must meet individual standards 1, 2, and 4 with or without reservations
Fails to meet WWC evidence standards if individual standards 1, 2, or 4 are not met
17
![Page 18: WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.](https://reader035.fdocuments.net/reader035/viewer/2022062721/56649f255503460f94c3c3b2/html5/thumbnails/18.jpg)
Reporting Requirement for Standard Errors Standard errors must reflect clustering at the
unit of assignment
Lee and Card (2008) – clustering issue applies when forcing variable is not truly continuous (“random misspecification error”)
This does not affect whether a study meets standards, but it does affect how its findings can be used
18