MNPS Academic Performance Framework 2011 - 2013

Paul Changas, Alan Coverstone and Christine StensonMetropolitan Nashville Public Schools (MNPS)

October 29, 2013

2

• Why the need for a Framework and how it will be used?

• Our process in building a Framework

• Components of our Framework – indicators and measures

• Next steps

• Lessons learned that might be helpful to other districts

• A framework template for other districts/schools

The MNPS FrameworkPresentation Overview

3

• To support efforts to raise student achievement

• To support the district’s accountability status

• To offer standardized accountability metrics to complement increased school-level autonomy

• To inform — but not determine — decisions regarding rewards, supports, and resource allocation for schools

• To provide school communities with a transparent set of indicators to understand school performance

The MNPS FrameworkPossible Framework Uses

4

• State absolute accountability system is primarily focused on districts rather than schools

• Key school relative accountability results reported every three years rather than annually

• State accountability is determined entirely by test scores – there are no school culture measures

• There is value in an overall performance index– While a single number cannot tell the whole story, a “bottom line” based upon

multiple measures is needed for making decisions– Transparency – outline exactly what measures will be included and how they will be

weighted– Manageability – quantity of data can be overwhelming

3 M’s of data use: make it Manageablemake it Meaningfulmake it Matter

Why the Need for a Framework?(Why Not Rely on State Accountability Results?)

5

State Accountability Flowchart

6

The MNPS FrameworkIndicators and Weighting

Academic Progress

50%

Attainment & College

Readiness 30%

Achievement Gap 5%

School Culture

15%

7

• Participants pair up

• Identify 3-5 key factors or indicators that you would include in a school performance system (academic, non-academic, school culture, etc.)

• How would you prioritize these factors (or should they be equally weighted)?

• Discuss for 5 minutes and then we will report out

What Factors Should be Included in School Evaluation?Audience Participation

8

The MNPS FrameworkK-8 Measures and Weighting

Minimum Maximum

TVAAS mean NCE gain -5 12 4% 25%

TCAP Mean achievement level increase as % of target -40 100 3% 25%

TCAP% of students scoring proficient 10 90 73% 15%

% of students projected from TVAAS to score 21 or above on ACT Composite

0 75 78% 15%

Achievement Gap

Index based upon gap in TCAP % proficient between subgroups

0 20 53% 5%5% Gap

Educator perceptions - TELL TN Survey Favorability 50% 100% 12% 5%

Parent perceptions NA NA NA 5%

Student perceptions - Tripod Favorability (Elementary) 55% 85%

Student perceptions - Tripod Favorability (Middle) 45% 75%

0 100 36% 100%Total Points (weighted composite)

Academic Progress

50% Progress

Attainment & College Readiness

30% Attainment

School Culture

15% Culture

0% 5%

Key Indicators Measure

Performance Scale Minimum & Maximum

% of Variation due to % Free or

Reduced Weight

9

The MNPS FrameworkHigh School Measures and Weighting

Minimum Maximum

TVAAS mean scale score gain -10 12 3% 25%

Mean achievement level increase as % of target -30 100 0% 25%

% of students scoring proficient 10 95 78% 10%

% of students scoring 21 or above on ACT Composite 0 95 89% 10%

% of students graduating on-time with a regular diploma 60 100 34% 10%

Achievement Gap

Index based upon gap in EOC % proficient between subgroups

0 20 58% 5%5% Gap

Educator perceptions - TELL TN Survey Favorability 50% 100% 0% 5%

Parent perceptions NA NA NA 5%

Student perceptions - Tripod Favorability 40% 75% 35% 5%

0 100 40% 100%Total Points (weighted composite)

Academic Progress

50% Progress

Attainment & College Readiness

30% Attainment

School Culture

15% Culture

Key Indicators Measure

Performance Scale Minimum & Maximum

% of Variation due to % Free or

Reduced Weight

10

The MNPS Framework The Academic Performance Scale

• Identify key performance measures

• Determine the weight of each measure

• Determine the performance scale of each measure

• Assign performance points to each school based upon position on performance scale

11

The MNPS Framework The Academic Performance Scale

Determine the performance scale of each measure

12

Performance Scale

Original NCE scale

-20 -10 0 10 20

-5 Performance Scale Minimum Scale Maximum

12

The MNPS FrameworkSchool Ratings for Growth

• Two measures (50% of Total Framework): Value Added and Mean Achievement Level Increase

• K-8 subjects: Math, Reading/LA, & Science

• HS subjects: Algebra I and II, English I, II, II, & Biology I

• Not highly correlated to socioeconomic status

13

The MNPS Framework Mean Achievement Level Increase

School A School B

2012 2013100

Per

cen

t of

Stu

den

ts

Advanced

10%Advanced

12%+2%90

Proficient

20%

30%Proficient

20%

32%80

+2%70

Basic

30%Basic

30%

60

50

+2%40

Below Basic

40%

Below Basic

38%

30

20

10

0

Overall 6% Improvement

2012 2013100

Per

cen

t of

Stu

den

ts

90

Advanced

10%-2%

Advanced

8%

Proficient

24%Proficient

20%

30%

Overall 4% Decline (despite increase in Profic/Adv)

32%80

+2%70

Basic

30%Basic

24%

60

50 -4%

40

Below Basic

44%

Below Basic

40%

30

20

10

0

14

The MNPS Framework Mean Achievement Level Increase Goal Example

Establish Proficiency Increase Goals Based Upon Prior-Year Results

Previous Year

Proficiency Increase Goal

Results if Goals Met

TOTAL = 0%+4%+9%+16% = 29%0%

Proficient

25%

Below Basic

24%

Basic

37%

= 0% x A = 0%

= 20% x P = 4%

= 30% x B = 9%

= 40% x BB = 16%

Advanced

14%

100%

Per

cen

t of

Stu

den

ts

Advanced (A)

10%90%

Proficient (P)

20%80%

70%

Basic (B)

30%

60%

50%

40%

Below Basic (BB)

40%

30%

20%

10%

15

The MNPS FrameworkK-8 Ratings for Achievement & College Readiness

• Two measures (30% of Total): Percent Proficient/Advanced and percent of 4th/8th grade students projected to score 21 or higher on the ACT

• Percent Proficient/Advanced includes Math (or Algebra I), Reading/LA, & Science

• These measures are correlated with socioeconomic status (SES), but schools often break the pattern

16

The MNPS FrameworkHigh School Ratings for Achievement & College Readiness

• Percent Proficient/Advanced includes Algebra I and II, English I, II, and III, and Biology I

• ACT Composite score of 21 or higher is required for the Hope Scholarship and is the average of the subject area college readiness benchmarks.

• These measures are correlated with Socioeconomic status, but schools often break the pattern.

17

The MNPS FrameworkAchievement Gap and Survey Data

• Gap Closure (5% of framework)

• K-8 subjects: Math, Reading/LA, & Science

• HS subjects: Algebra I and II, English I, II, II, & Biology I

• TELL and TRIPOD surveys each count as 5% of framework.

• Parent survey to be added

18

The MNPS FrameworkSchool Rankings Across Measures

19

The MNPS FrameworkReport Format

20

Charter School AccountabilityMNPS Commitments

1. Set and hold charter schools accountable to clear, measurable, and attainable academic, financial, and operational performance standards and targets;

2. Close schools that fail to meet performance standards and targets; and

3. Work proactively to identify and establish new, high quality charter schools to serve students who attend schools identified for closure.

Outcome-based Performance Management

Performance Contracts

(Predictable, enforceable)

Accountability-based

Interventions

(Transparent, balanced, comprehensive)

Consistent Communication

(Face validity, engagement)

District-Charter Collaboration Compact

Performance Management, Replication, and Closure (PMRC) Grant

MNPS Research and Assessment

Focus Groups Scorecard

Performance Frameworks Policies and Contracts

Technical Development Balanced Measures Broad Applicability

22

Outcome-based Performance ManagementHow will we use the APF?

• Publish Annual School Report Cards

• Shape Renewal Process

• Shape Recommendations each October

Academic Performance

Mean Achievement IncreaseTVAAS

TCAP (%PA)Achievement Gap

School Culture MeasuresACT (21+)

Overall Performance (APF)Year by year

Renewal Information

Projection and Review LevelRenewal Application Deadline

5-year Review Year

25

Outcome-based Performance ManagementHow will we use the APF?

• Publish Annual School Report Cards

• Shape Renewal Process

• Shape Recommendations each October

3-year status

Action Timeline

Renewal Review Level

 NA Simple Renewal

(Updated budget, plans, targets) 

Watch 2-yearsFull Renewal Review

(Full Renewal Application: Renewal Based on Best Interests Standard)

  1-yearConditional Renewal Review

(Renewal possible but unlikely; significant changes required; may recommend revocation depending on end of year results)

 Current

CycleNo Renewal

(May recommend revocation to take affect end of year)

27

Outcome-based Performance ManagementHow will we use the APF?

• Publish Annual School Report Cards

• Shape Renewal Process

• Shape Recommendations each October

School 2013 Status 3-Year Status Recommended ActionKIPP Academy Excelling Excelling Simple Renewal Review

Lead Academy MS Satisfactory Satisfactory None

LEAD Academy HS Satisfactory Satisfactory None

New Vision Academy Satisfactory Satisfactory None

Liberty Collegiate Acad Excelling 2-years None

STEM Prep Academy Excelling 2-years None

Drexel Prep School Target 2-years Notice: Revocation in 2014 Likely

Nashville Prep School Excelling 2-years None

East End Prep Insuffic Data 2-years None

Cameron College Prep Review 2-years None

Knowledge Academy Achieving 1-year None

Boy's Prep Target 1-year Notice: Revocation in 2014 Possible

Smithson-Craigh Acad Target Target Notice: Revocation in 2014 Likely

Smithson-Craigh MS Target Target Closed May 2013

29

• Participants pair up

• What types of decisions would you be comfortable making with a performance framework for your school(s)?

• How many years of data would you need to make these decisions?

• What additional information would you want in making decisions about school performance?

• Discuss for 5 minutes and then we will report out

What Decisions Would You Be Comfortable With?Audience Participation

30

The MNPS FrameworkNext Steps

• Show expanded data by individual year• Break out mean achievement level increase, TVAAS, and

gap calculations by subject• Break out key results by subgroup• Provide professional development on the Framework• Expand documentation• Utilize results in evaluation of initiatives and in identifying

schools needing support and resources

31

The MNPS FrameworkLessons Learned

• Include stakeholders in development• Transparency is critical• Flexibility is essential as standards, assessments and policies

change, but the basic principles we value and include in the Framework should hold up over time

• Construct the Framework in such a way that the top performance category is within reach of any school, regardless of socioeconomic factors

• Performance measures can vary significantly from year to year, so multiple years of data should be utilized in critical decisions

• Presentation format is important

32

The MNPS FrameworkPerformance Framework Template

MeasureScale

MinimumScale

MaximumReverse

Scale?Weight (0-100)

School Value

% of Possible

PointsUnadjusted # of Points

Adjusted# of Points

1 TVAAS Mean Scale Score Gain -5 12 No 25 2.5 44.1% 11.03 20.052 TCAP % Proficient/Advanced 10 90 No 15 48 47.5% 7.13 12.953 Achievement Gap Index 20 0 Yes 5 7.2 64.0% 3.20 5.824 Tripod Survey Favorability 0.55 0.85 No 5 0.68 43.3% 2.17 3.945 TELL TN Survey Favorability 0.5 1 No 5 0.77 54.0% 2.70 4.916 0 100 No 0.0% 0.00 0.007 0 100 No 0.0% 0.00 0.008 0 100 No 0.0% 0.00 0.009 0 100 No 0.0% 0.00 0.00

10 0 100 No 0.0% 0.00 0.0011 0 100 No 0.0% 0.00 0.0012 0 100 No 0.0% 0.00 0.0013 0 100 No 0.0% 0.00 0.0014 0 100 No 0.0% 0.00 0.0015 0 100 No 0.0% 0.00 0.0016 0 100 No 0.0% 0.00 0.0017 0 100 No 0.0% 0.00 0.0018 0 100 No 0.0% 0.00 0.0019 0 100 No 0.0% 0.00 0.0020 0 100 No 0.0% 0.00 0.00

TOTAL 55 26.22 47.67

Academic Performance Framework Template

33

Each Category of Data Requires the Following Decisions:

Which Measure

Subject Areas

Years of Data

Unit (e.g. School)

Performance Thresholds

Weighting/ Points

For Example: Value Added• Should Math, Reading/LA, Science be included?• Should Math, Reading/LA, Science be weighted equally and be combined into a single score?• Should there be discrete score categories (e.g. 1-5), or a linear transformation of the scores such that schools earn from 0 to 100 percent of possible points, or should we use a method that takes into account the fact that most schools’ scores are going to be clustered around the average?• Should the categories be discrete or continuous, do we base them on average growth from 2012 to 2013, or take other years of growth into consideration? If we use two years of data, we get a bigger range of possible scores. • Do we set our evaluation based on what has been average in the school district and the state or based on the growth standard (i.e. zero growth is average)? If we use the growth standard, we may fall behind the state. If we use actual average NCE gains, we will be rating schools on a very different basis than the state does in its report card and on the TVAAS site (for grades 3-8 only).

The MNPS FrameworkDecision-making Process for Each Indicator

34

The MNPS Framework

Questions?