Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia...
-
Upload
gwenda-curtis -
Category
Documents
-
view
213 -
download
0
Transcript of Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia...
![Page 1: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/1.jpg)
Copyright © 2015 American Institutes for Research. All rights reserved.
Elizabeth Spier, PhD Johannes Bos, PhDPrincipal Researcher Senior Vice President
FAST in Philadelphia
SEPTEMBER 2015
Keeping an RCT Relevant When Program Participation is Low
![Page 2: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/2.jpg)
2
The Intervention
![Page 3: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/3.jpg)
Families and Schools Together (FAST) is a school-based initiative intended to improve children’s academic and social outcomes.
For this study, FAST is being implemented in 30 persistently-low-performing schools in Philadelphia.
The School District of Philadelphia is the eighth largest in the United States, and serves many ethnically and linguistically diverse, high-poverty communities.
Families and Schools Together
3
![Page 4: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/4.jpg)
• FAST invites students and their families into the school for a series of after-school sessions focused on improving connections within families, among families, and between families and schools
Objectives include:• Better communications and less stress at home• Better social support for families• Better behavior by the students (at home and in school)• More positive school-home engagement and collaboration
How FAST Works
4
![Page 5: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/5.jpg)
• Investment in Innovation (i3) grant from U.S. Department of Education, focused on school turnaround.
• School turnaround is achieved by getting and keeping children on track academically and socially.
• FAST was offered universally to all families of incoming kindergarten (grade 0) students.
• A relatively high level of FAST participation is necessary to improve outcomes across the school cohort (not just for individual students).
The Current Project
5
![Page 6: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/6.jpg)
FAST Logic Model
6
FAST
Parent-Child Relationship
Building
Parent-School Engagement
Improved Early
Learning
Improved Home
Environment
Parental Social Support
Improved Child
Behavior
Improved Classroom
Climate
Improved School Climate
School Turnaround
![Page 7: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/7.jpg)
The Evaluation
7
![Page 8: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/8.jpg)
Original Sampling Design
8
Random Selection
Random Assignment
60 Eligible Schools
60 Schools
30 Schools FAST 2013
12 Students Cohort 1
12 Students Cohort 2
30 Schools FAST 2016
12 Students Cohort 1
12 Students Cohort 2
![Page 9: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/9.jpg)
The sampling design was based on these assumptions:• If FAST was offered to all incoming students and their
families, most would choose to participate.• Families who started to participate in FAST would
continue to do so, so that most students’ families (at least 60%) would attend 4 or more sessions.
• If most students in the cohort participated in FAST, then impacts (if any) would be evident at the cohort level.
Assumptions
9
![Page 10: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/10.jpg)
Reality
10
17%
43%
40%
Completers
Non-Completers
Not expected to complete FAST
![Page 11: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/11.jpg)
Sampling Design Amplifies Consequences
11
Kindergarten Class
Kindergarten Class
![Page 12: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/12.jpg)
Sampling Design Amplifies Consequences
12
Kindergarten Classroom
Expected Participants
Kindergarten Class
Reality
![Page 13: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/13.jpg)
Sampling Design Amplifies Consequences
13
Kindergarten Classroom
Expected ParticipantsStudy
Sample
Kindergarten Classroom
RealityStudy
Sample
![Page 14: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/14.jpg)
Sampling Design Amplifies Consequences
14
Kindergarten ClassroomStudy Sample
Study Sample
![Page 15: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/15.jpg)
Alternative Supplemental Design
15
Kindergarten Classroom Enhanced
study sample
Study Sample
![Page 16: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/16.jpg)
• We are keeping the original RCT intact but redirected some study resources to a supplemental study to measure the effect of “treatment on the treated”
• For this study, we are using a two-level propensity score matching design that maximizes the contrast between participants and non-participants
• Note that this supplemental study answers somewhat different research questions!
Alternative Supplemental Design
16
![Page 17: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/17.jpg)
Multi-Level Mahalanobis Matching
17
Treatment Control
![Page 18: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/18.jpg)
Multi-Level Mahalanobis Matching
18
1 2
3
4
5
6
7 8
8 4
7
5
1
6 2
3
Treatment ControlSTEP 1Match 8
treatment schools with
highest FAST
participation rates to
comparable control schools
![Page 19: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/19.jpg)
Multi-Level Mahalanobis Matching
19
1 2 3 4
5 6 7 8
3
4
5
6
7 8
1 2 3 4
5 6 7 8
5
1
6 2
3
Treatment ControlSTEP 2Collect
baseline data on all
kindergarten students and their families in these 16
schools
![Page 20: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/20.jpg)
Multi-Level Mahalanobis Matching
20
Treatment ControlSTEP 3Identify
participating children in treatment
schools and match them
to similar children in
control schools
![Page 21: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/21.jpg)
Multi-Level Mahalanobis Matching
21
Treatment ControlSTEP 4
Use resulting sample to estimate impact of
participating in FAST
(treatment on the treated)
![Page 22: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/22.jpg)
Closing Thoughts
22
• When planning RCT with non-universal data collection, consider take-up and participation as a factor.
• Design studies to capture both individual-level “treatment on the treated” effects and school or classroom-level effects, ideally with comparable rigor for both.
• When possible, collect universal baseline data even in cases where you plan to use random subsamples for follow-up data collection (increases flexibility).
• Please stay tuned for impact results!
![Page 23: Elizabeth Spier, PhDJohannes Bos, PhD Principal ResearcherSenior Vice President FAST in Philadelphia SEPTEMBER 2015 Copyright © 2015 American Institutes.](https://reader035.fdocuments.net/reader035/viewer/2022070413/5697bfd81a28abf838caef57/html5/thumbnails/23.jpg)
Elizabeth Spier, PhD Johannes Bos, PhD650-843-8226 [email protected] [email protected]
2800 Campus Drive, Suite 200San Mateo, CA 94403General Information: 650-843-8100TTY: 650-493-2209www.air.org
23