Dr. Bonnie J. Faddis & Dr. Margaret Beam RMC Research Fidelity of Implementation and Program Impact.
-
Upload
lee-farmer -
Category
Documents
-
view
221 -
download
4
Transcript of Dr. Bonnie J. Faddis & Dr. Margaret Beam RMC Research Fidelity of Implementation and Program Impact.
2
PPS Striving Readers Partners
• Grantee: Portland Public Schools
• Professional Developer: University of Kansas, Center for Research on Learning
• Evaluator: RMC Research
3
Targeted Intervention Overview
• Goals:– To increase reading achievement
– To increase reading motivation
• Intervention features: – Xtreme Reading curriculum
– Small class size (15 students/teacher)
– Ongoing professional development
4
Eligibility for Intervention
• Students enrolled in Grades 7–10
• Students reading 2 or more years below grade level on the Oregon State Assessment Test (OSAT)
• Students without OSAT scores who are reading 2 or more years below grade level on the GRADE test
5
Intervention Characteristics
• 9 schools – 5 middle schools– 4 high schools
• 18 targeted intervention teachers (2 per school)
Year 1
• 9 schools – 1 middle school– 2 K–8 schools– 1 6–9 girls’ school– 1 8th grade academy– 4 high schools
• 18 targeted intervention teachers
Year 2
6
Intervention Characteristics
Year 3
• 10 schools– 2 middle schools– 3 K-8 schools– 1 6-10 girls’ school– 4 high schools
• 14 targeted intervention teachers
7
Intervention Staffing Plan
• 25-30 students per grade level assigned to intervention class at each school
• Intervention class paired with language arts or social studies class in high school
• Intervention class integrated with LA/SS block in middle school
• 2 teachers co-taught intervention class and LA/SS class (Years 1-2)
8
Counterfactual
• Xtreme Reading taught during LA/SS block
• Control group students received LA/SS instruction with higher student-teacher ratios
Middle Schools
• Xtreme Reading was an elective course
• Control group students enrolled in a range of other electives across content areas
High Schools
9
Xtreme Reading Model
Metacognition
LINCS Vocabulary Strategy Learning new ways to remember the meaning of vocabulary
Word Mapping Strategy Learning new ways to remember the meaning of vocabulary
Word Identification Strategy Learning how to pronounce multisyllabic words
Self-Questioning Strategy Learning to ask yourself questions, make predictions, and talk about answers as you read
Visual Imagery Strategy Learning to make pictures in your mind while reading a passage
Paraphrasing Strategy Learning to put main ideas and details in your own words
Inference Strategy Learning to ask and answer thoughtful questions as you read, infer, and predict information
Motivation
Possible Selves Increasing student motivation by thinking about important goals for the future
Class Management
Xpect to ACHIEVE Understanding and following clear guidelines that support a successful learning community
Community Learning Skills
The SCORE Skills Social skills for cooperative groups
Talking Together How to participate respectfully in class discussions
10
8 Stages of Instruction
• Describe rationale and strategy steps• Model strategy through think-aloud• Verbally rehearse strategy steps• Guided reading practice• Paired reading practice with fluency tests• Independent practice w/comprehension check• Apply strategy during oral reading with
teacher• Integrate strategy with text from other classes
11
Implementation Results
• Professional Development
• Classroom Implementation
• Teacher Buy-In
• Factors that Facilitated Implementation
• Barriers to Implementation
12
Professional Development Participation
• Group session attendance > 75%– Year 1: 40% of middle school and 25% of high
school Xtreme teachers
– Year 2: 89% of middle school and 75% of high school Xtreme teachers
– Year 3: 100% of middle school and 71% of high school Xtreme teachers
• Added in Year 2 & 3: – Monthly meetings of Xtreme teachers
13
Classroom Implementation
School
Average Fidelity
Year 1 Year 2 Year 3
Middle Schools
Alpha 92% 89% 79%*
Beta 96% 90% 92%
Gamma 72% 84% 93%
Delta 93% 91% 94%*
Epsilon 94% 88% —
Phi — — 95%
Zeta — — 80%*
High Schools
Kappa 83% 63% 92%
Lambda 84% 58% 82%
Sigma 82% 78% 74%*
Theta 45% —* 80%*
14
Teacher Buy-In
Professional Development &
Support*
Perceived Program
Effectiveness**
Middle High Middle High
Year 1 4.2 3.6 4.1 3.6
Year 2 4.2 3.2 4.0 3.7
Year 3 4.4 4.0 4.6 4.0
* Rating Scale: 1 = Strongly disagree; 5 = Strongly agree** Rating Scale: 1 = Not at all helpful; 5 = Very helpful
15
Factors that Facilitated Implementation
• Xtreme teachers liked the curriculum• Students liked the books• Teachers liked the professional
development• Curriculum pacing schedules improved• 80% of Year 2 teachers had experience• 62% of Year 3 teachers had experience
16
Barriers to Implementation
• Organization of Xtreme materials was confusing
• Teacher skills with low achievers• Changes in middle school configurations
reduced target population
• Small school organization in 2 high schools created scheduling problems
• High school counselors resistant
17
Impact Evaluation
• Questions– Was the Striving Readers Xtreme
intervention effective in improving students’ reading achievement?
– Were the effects of the Striving Readers Xtreme intervention similar for middle schools and high schools?
18
Impact Evaluation
• Random AssignmentEligible students were randomly assigned to treatment or control (stratified by school and grade level)
• Analytic DesignIntent-to-treat statistical model using 2-level HLM (students clustered within schools)
19
Impact Evaluation
• Outcome MeasureSpring GRADE test normal curve equivalent (NCE) scores
• Covariates– Fall GRADE pretest normal curve equivalent
(NCE) scores– Cohort– Ethnicity (Black, Hispanic)– English language proficiency– Grade level
20
Statistical Power
• 2-level HLM cluster randomized trial
• N = 1,273
• Power = 80%
• Alpha = .05
• Minimum detectable effect (MDE) = .27
21
Random Assignment
Striving Readers Enrollment
Met eligibility criteria n = 2,438
Randomized
Treatment Group
Allocated to intervention n = 1,223
Excluded from study n = 296
Lost to follow-up n = 328
Analyzed n = 599
Allocation
Ineligibility
Follow-Up
Control Group
Allocated to intervention n = 1,215
Excluded from study n = 267
Lost to follow-up n = 274
Analyzed n = 674
Analyzed n = 674
Analysis
23
Student Characteristics
CharacteristicTreatment
PercentControl Percent
Gender
Male 53 53
Female 47 47
Ethnicity
White 25 25
American Indian 2 2
Hispanic 30 31
African American 28 29
Asian 15 12
Special Education Services 25 25
ELL Services 27 25
Note. Treatment total N = 599; Control total N = 674.
24
Impact Results
Estimated Impact
Group nImpact
() S.E.Effect Size
p-value
Overall 849 2.58 0.79 .15 .002
Middle School
443 4.84 0.90 .29 .000
High School 406 0.16 1.31 .01 .901
25
Impact Results
• Additional Finding
Significant school level variability in treatment effects in both the overall analysis and in the middle school analysis
26
Effects of Implementation
• Question
To what extent do teacher level variables explain school level variability in treatment effects?
27
Analytic Design
• Statistical model using 2-level HLM (students clustered within schools)
• Teacher level variables aggregated up to school level
• Separate models by implementation year
28
Level 2 Variables
• Percent fidelity of implementation
• Percent of professional development training attended
• Years of teaching experience
29
Results
• Fidelity of implementation significantly contributed to school level variability in treatment effects in Years 1 and 3, but not in Year 2