Value-added Accountability for Achievement in Minneapolis Schools and Classrooms
-
Upload
grant-williamson -
Category
Documents
-
view
25 -
download
3
description
Transcript of Value-added Accountability for Achievement in Minneapolis Schools and Classrooms
Value-added Accountability for Value-added Accountability for Achievement in Minneapolis Achievement in Minneapolis Schools and ClassroomsSchools and Classrooms
Minneapolis Public SchoolsMinneapolis Public Schools
December, 2000December, 2000
www.mpls.k12.mn.us/REA/www.mpls.k12.mn.us/REA/
Minneapolis Pubic SchoolsMinneapolis Pubic SchoolsMeasuring Up Against High StandardsMeasuring Up Against High Standards MPS developed standards-based reading and math
tests - Northwest Achievement Levels Tests (NALT). All students participate Sensitive to growth Predictive of high stakes outcomes
Developed indicators in concert with stakeholders who are internal and external to the district.
Indicators of school performance are based on multiple indicators. 33 for elementary and middle schools 24 for high schools
Standards Based means:Standards Based means: We use a mutually agreed upon rubric for We use a mutually agreed upon rubric for
each indicator of school performance.each indicator of school performance. Rubric scoring on a “1” to “5” scale.Rubric scoring on a “1” to “5” scale.
““3” Current standard3” Current standard ““5” Long term goal5” Long term goal
Data are reported by site, by indicator.Data are reported by site, by indicator. Community reports list percent of indicators Community reports list percent of indicators
at standard (3 or better) “Measuring Up.”at standard (3 or better) “Measuring Up.”
Selecting a unit of AnalysisSelecting a unit of Analysis
Minneapolis Public Schools in concert Minneapolis Public Schools in concert with multiple stakeholders agreed upon with multiple stakeholders agreed upon a matrix of indicators.a matrix of indicators. School PersonnelSchool Personnel Community input (e.g. NAACP)Community input (e.g. NAACP) Chamber of CommerceChamber of Commerce Minneapolis FoundationMinneapolis Foundation FamiliesFamilies
Criteria for IndicatorsCriteria for Indicators
Meyer (1996)Meyer (1996) Indicators must measure things that matter Indicators must measure things that matter
or are valued by society (MCA, MBST).or are valued by society (MCA, MBST). Indicators must be able to avoid being Indicators must be able to avoid being
“corrupted.”“corrupted.” inclusion v. exclusion criteriainclusion v. exclusion criteria
Indicators must be able to measure the Indicators must be able to measure the unique contribution schools “add” for each unique contribution schools “add” for each measured outcome.measured outcome.
Stakeholders agreed on Stakeholders agreed on using…using… Multiple indicators of school Multiple indicators of school
performance.performance. See chart of indicators from Appendix.See chart of indicators from Appendix. These indicators include:These indicators include:
Current level of performanceCurrent level of performance cross-cohort cross-cohort Gain and value-added measuresGain and value-added measures
Elementary & Middle School Elementary & Middle School QPA IndicatorsQPA Indicators
Type Level Cross-Cohort Gain – ValueAdded
AchievementTests (24)
State Tests(5)
State Tests(5)
State Tests,Oral Reading,
NALT (14)
Correlatedwith
Achievement(9)
Attendance,Suspensions,
Safety,Respect,
Gifted Programs,Summer SchoolParticipation (7)
Attendance,Suspensions,
(2)
Fatal Flaws of a level or means Fatal Flaws of a level or means approach.approach.
Mean test performance is Mean test performance is contaminated contaminated by by factors other than school performance.factors other than school performance.
Mean performance is out of date, and does Mean performance is out of date, and does not address cumulative, compounding effects not address cumulative, compounding effects of previous instruction.of previous instruction.
Mean performance is affected by mobility.Mean performance is affected by mobility. Mean performance interferes with localizing Mean performance interferes with localizing
performance.performance.
Distribution of school average 1999 NALT Distribution of school average 1999 NALT mathematics scores by free or reduced price mathematics scores by free or reduced price
lunch percentageslunch percentages
R2 = 0.7901
30
40
50
60
70
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Percent free or reduced price lunch
NA
LT
Mat
h A
vera
ge N
CE
School P = 44.4 nce
Distribution of school average mathematics gain Distribution of school average mathematics gain scores on the 1999 NALT by free or reduced price scores on the 1999 NALT by free or reduced price
lunch percentages.lunch percentages.
R2 = 0.0074
-3-2-10123456789
10
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Percent free or reduced price lunch
NA
LT
Mat
h N
CE
Gai
n
School P = 6.3 nce gain
Cross-Cohort AnalysisCross-Cohort AnalysisDistrict LevelDistrict Level
Percent of MPS Students Passing Minnesota Mathematics Basic Standard Graduation Test in 8th Grade
28%21%
38%
22%
40%
55%
29%
72%74%
27%
45%
19%
33%
74%
24%
0%
20%
40%
60%
80%
100%
AmericanIndian
AfricanAmerican
AsianAmerican
Hispanic White
1998
1999
2000
Standard
The problem with cross-cohort The problem with cross-cohort analysis at the school levelanalysis at the school level
1200
1240
1280
1320
1360
1400
1440
1998 1999 2000 2001 2002 2003
Mean scale score
Grade 3 Reading
Grade 3 Math
Grade 5 Reading
Grade 5 Math
Expected Growth= 26 points/yearin 3rd grade reading
Look at the “bounce” incross-cohort analysis
Northwest Achievement Levels Test Validity Northwest Achievement Levels Test Validity Coefficients with Minnesota State TestsCoefficients with Minnesota State Tests
Grade/Measure Grade 3ReadingMCA
Grade 3MathMCA
Grade 5ReadingMCA
Grade 5MathMCA
Grade 8ReadingMBST
Grade 8 MathMBST
Grade 2 NALTa (n/a) .80(n= 3219)
Grade 3 NALTb .87(n=3785)
.87(n=3640)
Grade 4 NALTa .85(n= 3153)
.86(n=3097)
Grade 5 NALTb .88(n=3533)
.89(n=3484)
Grade 7 NALTc .82(n= 2608)
.88(n= 2631)
aNALT April, 1998 correlation with MCA March, 1999 for same subject area (predictive validity)bNALT April, 1999 correlation with MCA March, 1999 for same subject area (concurrent validity)cNALT April, 1998 correlation with MBST February, 1999 for same subject area (predictive validity)
Linkage with State StandardsLinkage with State Standards
NALT Scale Score Which Predicts Passage of theMinnesota Basic Standards Test (MBST) in 8th Grade
Grade Reading Scale Score Reading Scale Score
3 190 1994 197 2065 204 2146 209 2187 213 2258 218 232
True gain model: Growth True gain model: Growth CurvesCurves
Class of 2005 Longitudinal Reading Gains
190
200
210
220
230
Grade 3
Grade 4
Grade 5
Grade 6
Grade 7
Grade 8
Grade 9
Grade 10
Grade 11
Grade 12
NA
LT
Sc
ale
Sc
ore
(R
as
ch
Mo
de
l)
High Standard
Basic Standard
Class of 2005 (n=2006)
True Gain model: Percent of True Gain model: Percent of Students Making One Year GainsStudents Making One Year Gains
Percent of Students Making One Year Growth or More on NALT Reading and Math by Program
55% 53%47%
56%53% 51%47%
54%
0%
20%
40%
60%
80%
100%
All ELL SpecialEducation
Mainstream
Per
cen
tage
of
Stu
den
ts
Reading
Math
District Standard
1998 1999Top 1/5 73.70 77.00 226 StudentsMedian 52.10 57.25Bottom 1/5 31.50 39.80
1998-99 EmersonMath Concepts 20/20 Analysis
10
20
30
40
50
60
70
80
90
1998 1999
Year
Norm
al C
urv
e E
quiv
ale
nt
(NC
E)
Top 1/5
Median
Bottom 1/531st percentile54th percentile
64th percentile
87th percentile
90th percentile
19th percentile
1998 1999Top 1/5 73.18 78.20 168 StudentsMedian 47.65 55.30Bottom 1/5 29.90 35.10
1998-99 EmersonReading Comprehension 20/20 Analysis
10
20
30
40
50
60
70
80
90
1998 1999Year
Norm
al
Curv
e E
quiv
ale
nt
(NC
E)
Top 1/5
Median
Bottom 1/5
24th percentile46th percentile
60th percentile
86th percentile
91st percentile
17th percentile
Value-added ModelValue-added Model Primary question is thisPrimary question is this, “, “How can an equitable How can an equitable
accountability system be put in place when there is a accountability system be put in place when there is a large degree of variability in the way student large degree of variability in the way student characteristics are distributed among our schoolscharacteristics are distributed among our schools?”?” English language learnersEnglish language learners Special education programsSpecial education programs Race/ethnicityRace/ethnicity PovertyPoverty
MPS relies on an empirical- and literature-based MPS relies on an empirical- and literature-based model to make predictions, not set expectations! model to make predictions, not set expectations!
Schools are acknowledged/rewarded for “Schools are acknowledged/rewarded for “Beating Beating the Oddsthe Odds.”.”
Value-added Teacher EffectsValue-added Teacher Effects
Post-test reading score=Post-test reading score= Pretest reading scorePretest reading score + Free or reduced price lunch+ Free or reduced price lunch + Racial/ethnic code+ Racial/ethnic code + Neighborhood poverty concentration+ Neighborhood poverty concentration + Lives with both parents code+ Lives with both parents code + Limited English Proficiency status+ Limited English Proficiency status + Special Education status+ Special Education status + Teacher effects+ Teacher effects
Teachers who “beat the odds” in Teachers who “beat the odds” in second grade reading reported: second grade reading reported:
more use of small group instructionmore use of small group instruction more development of word attack skillsmore development of word attack skills more individual student oral readingmore individual student oral reading more guidance during initial practicemore guidance during initial practice more explicit and direct phonics more explicit and direct phonics
instructioninstruction more use of systematic motivation more use of systematic motivation
strategiesstrategies