College Success Academy: Launching a New Program with Research and Evaluation Partners
-
Upload
national-partnership-for-educational-access -
Category
Documents
-
view
174 -
download
1
description
Transcript of College Success Academy: Launching a New Program with Research and Evaluation Partners
Launching a new program with research and evaluation partners
NPEA Conference April 11, 2013
Agenda
• Welcome and introductions• How we got here• Program design: the scorecard• Planning evaluation activities• What we learned• The next chapter: measuring non-cognitive
Who we are
• Robert Theaker, Senior Research AssociateThe Kingsbury Center, Northwest Evaluation Association
• Roblyn Brigham PhD, Managing PartnerBrigham Nahas Research Associates
• Yully Cha, Chief Program OfficerThe Steppingstone Foundation
What we believe about our work
• Evaluate early and often
• Culture matters
• Broad view of what we mean by “data”
• Findings inspire action
• Students front and center
How we got here
• Environmental context– Summer learning loss and the achievement gap– College access to persistence/graduation
• The Steppingstone Foundation’s 2009 strategic plan– The public school venture
Program design1. Academic achievement Center for Higher Education Studies, UC Berkeley;
Cliff Adelman, U.S. Dept of Ed; National Center for Educational Accountability
2. Socio-emotional competency Malecki et al (Measuring Perceived Social Support: Development of the Child and Adolescent Social Support Scale)
• Adult relationships Learning First Alliance (Every Child Learning: Safe and Supportive Schools)
• Self-efficacy Robbins et al, (Do Psychosocial and Study Skill Factors Predict College Outcomes? A Meta-Analysis)
3. Positive behavior Balfanz et al (Preventing Student Disengagement and Keeping Students on the Graduation Path in Urban Middle Grades Schools)
• Attendance• School disciplinary action
4. College awareness The Bridgespan Group (Reclaiming the American Dream); Southern Regional Education Board (Middle Grades to HS: Mending a Weak Link; Choy, U.S. Dept of Ed (Students Whose Parents Did Not Go to College: Postsecondary Access, Enrollment and Persistence)
Academic achievement• The search for the right tool– Summative– Formative– Measure summer learning– And we want to compare against national norms
Measures of Academic Progress (MAP)
National Map
• Partners in 50 states• Over 5000 Partner Districts• Over 6 million students assessed • Partners in 100 foreign countries
Equal interval Linked to
curriculum Achievement scale Cross graded Shows growth Greater score
precision Like an academic
ruler Beginning Reading
Adult Reading
RIT SCALE
X x
x
x x
Xx
xx
X
x x
x
x
x
184
207
232
NWEA Uses a RIT Scale (Rasch Unit)
Daniel
Devon
Grace
High Performing Example
Grace’s Test Pattern
Planning with external evaluator
Logic model:Articulate “the model”Define outcomes and
sequenceBuild consensus around
data
Implement observations, focus groups, and
interviewsInternal team debrief
sessionsBuild SPSS database
Mid-year observations and report
Continue evaluation activities
Three-year analysis and report
Data collection plan
• Qualitative Data – Observations, interviews, focus groups– Program staff, teachers, parents, tutors, and
students• Quantitative Data– Scorecard and MAP– Non-cognitive measures– Surveys to capture perspective of parents
• Process of sharing what we learn
What we learned: qualitative
Strengths
• Program administration is strong
• The program “culture” is taking root
• Scholars’ enthusiasm for the academics
Challenges
• Demanding job for teachers/staff
• Culture needs to deepen to transform
• Defining who to serve/who can best benefit from the program
Our response
• Change schedule and temperature
• Admission info and interview sessions
• IEP info collected in admission; improve faculty orientation
What we learned: quantitative
English Language Arts Math
Score SGP* A/P** Score SGP* A/P**
Grade 4 (2011)
230 n/a 13 230 n/a 11
Grade 5 (2012)
232 55 18 238 65 19
Class 1 Class 2
Summer 2011 93% n/a
Academic Year 2011-2012
87% n/a
Summer 2012 92% 94%
Program attendance: average daily attendance
MCAS
Average GPA
Average days missed from school
Grade 4 (2011) 2.7 8Grade 5 (2012) 2.8 6
Class 1 Class 2
Summer 2011 42/46 n/a
Academic Year 2011-2012
35/42 n/a
Summer 2012 29/35 = 63% 26/46 = 78%
Retention43% of attrition due to mobility
Grades
What we learned: measuring academic impact
• Year one – Correcting mistakes in how test is administered– Learning how to read and report the results
• Year two– Summer learning impact and school year effects– English Language Learners– Boys
Scores and percentiles
How we create a Virtual Comparison Group (VCG)
19
We identify included studentsIdentify all matching students from GRDGrade, Subject, Starting Achievement, School Income, Urban vs. Rural Classification, etc.Randomly select comparison group
20
Compare your
student’s growth to
similar students in
similar schools
Virtual Comparison Groups(VCGs)
Next chapter: measuring non-cognitive
• Virtual Comparison Group (VCG) analysis• External evaluator year-three report• Non-cognitive assessments– Holistic Student Assessment– Survey of After-School Youth Outcomes (SAYO)