2013-14 Median Student Growth Percentile Reports: An Overview

January 2015

2

Bottom Line Up Front

All districts received secure access to their 2013-14 teacher and principal/AP/VP Median Student Growth Percentile (mSGP) data on January 8, 2015.

• Teachers and school leaders have the highest impact on student achievement of any in-school factor.

• NJDOE has worked with NJ educators in taking a long and thoughtful approach to implementing both evaluations and mSGP.

• mSGP data is an important part, but only one part of an educator’s evaluation. These scores will be used to finalize 2013-14 evaluations and to inform educators’ ongoing professional development.

By statute, mSGPs (like all aspects of an individual’s evaluation) are confidential and should not be shared publicly.

3

Context for SGPs in New Jersey

• Student Growth Percentile (SGP) methodology is not new; SGP was adopted in the 2009-2010 school year and first used for school accountability in the 2011-2012 annual performance reports.

• TEACHNJ Act requires use of at least one measure of student growth in evaluations (18A:6-123: “Standardized assessments shall be used as a measure of student progress…”).– AchieveNJ incorporates 2 measures of growth for teachers:

mSGP and SGOs and 3 measures for principals: mSGP, SGO Average, Administrator Goals.

• NJDOE has been building capacity to share mSGP for over 4 years, including distribution of 2011-12 teacher data to pilot districts and 2012-13 teacher data to all districts.

• mSGP data is derived from roster and staff submission data provided by districts through a submission process in place for many years.

4

Timeline of SGP Development in New Jersey

A thoughtful, multi-year approach to ensure data is accurate and usable

2014 20152011 2012 20132010

Federal Mandate for Stimulus Funds: States Must Calculate “Student Growth”; Link Teachers to Students

Student SGPs Provided to All Districts in NJ SMART

NJ Adopts SGP Methodology for Calculating Student Growth

SGP Training Begins for Districts; SGP Video Released

TEACHNJ Act Passed; Growth

Measures Required for Evaluation

District SGP Profile Reports Deployed via NJ SMART

School SGPs Used in School Performance Reports per NJ’s Federal ESEA Waiver

2012-13 Teacher mSGP Reports Provided to All Districts for Learning Purposes and Data Preview

2011-12 Teacher Median SGP (mSGP) Reports Provided to Pilot Districts for Learning Purposes

2013-14 Teacher & Principal mSGP Reports Provided to All Districts for Use in Evaluations; Score Verification Process Announced

Evaluation Pilot Advisory Committee

Provides Feedback on Usefulness of

SGP Data2013-14 mSGP Score Verification & Certification Process Completed by Districts

5

2013-14 mSGP Data

• The 2013-14 mSGP data counts for 30% of qualifying teachers’ and 20 or 30% of qualifying principals’ 2013-14 evaluations.

• Evaluation data of a particular employee shall be confidential in accordance with the TEACHNJ Act and N.J.S.A. 18A:6-120.d and 121.d.

– Educator mSGP data should be handled in the secure manner one would treat, handle, and store any part of a confidential personnel record and should not be released to the public.

• While a dry run for teacher mSGP data was conducted last year to improve roster verification processes, if educators identify a problem with the 2013-14 mSGP score, the Department is offering options for addressing the issue.

6

Student Growth PercentileStudent Growth Objectives

mSGP Qualification and Weighting: Teachers

1. Teach a 4th–8th grade Math or Language Arts Literacy (LAL) class, and

2. Be the teacher of record for at least 60% of the course prior to the NJ ASK assessment, and

3. Have at least 20 students with valid SGP scores who are enrolled in the class for at least 70% of the school year before they take the NJ ASK.

To receive an mSGP score, teachers must:

Teacher Practice

55%15%

30%

2013–14

7

mSGP Qualification and Weighting: Principals

1. Be assigned to schools attended by more than 20 separate students who took the 4th- to 8th-grade Mathematics or Language Arts NJ ASK; and

2. Be assigned to qualifying students reported as having attended the school for one full year prior to the administration of the NJ ASK, according to the data submitted by the district in its State Submission.

To receive an mSGP score, principals/APs/VPs must:

Principal PracticeEvaluation Leadership

SGO AverageAdministrator GoalsSchool SGP

30%

20%10%

20%

20%

Single-Grade SGP Principals/APs/VPs(1 SGP grade in school)

30%

20%

10%

10%

30%

Multi-Grade SGP Principals/APs/VPs

(2 or more SGP grades in school)

2013-14

8

Individual mSGP Report

Individual mSGP reports provide:

• Background information about SGPs and mSGPs

• The educator’s mSGP score for 2013-14; Chart depicting an

example of a teacher’s score is displayed below

• Suggestions for how to interpret and utilize the data, as

well as access to additional resources

TEACHER MSGP

# STUDENTS ASSIGNED

MSGP SCORE(1.0-4.0)

Language Arts Literacy 46 22

Mathematics 68 22Overall 52 22 3.0

9

Addressing Data Issues

• The accuracy of mSGP Data is dependent on course roster and staff submission data by the school district.

• Districts may expect to receive mSGP scores for educators who ended up not qualifying – and vice versa.

• Incorrect district data submissions may have led to errors in mSGP scores for individual educators.

• Guided by feedback from last year’s practice release, the Department has created mSGP score verification and correction procedures. • Depending on the issue and the district’s ability to document evidence

for a given error, the mSGP score may be recalculated or removed from an educator’s summative rating.

• Detailed guidance has been posted here.

10

Potential Data Issues

• Educators expecting but not receiving the mSGP score– Option A: For a teacher with 2 SGOs for 2013-14 OR for a

principal/AP/VP, recalculate summative rating using other component scores

– Option B: For a teacher with 1 SGO, eliminate summative rating

• Educators seeking to correct the mSGP score– Option A: Provide documentation of errors, correct mSGP

score, and recalculate summative rating– Option B: Remove mSGP score for a teacher with 2 SGOs OR

for a principal/AP/VP, and recalculate summative rating– Option C: Remove mSGP score for a teacher with 1 SGO,

eliminate summative rating.Note: State law requires multiple measures of student achievement; thus, if district data errors result in the need to negate the mSGP score for a teacher – and that teacher did not set 2 SGOs – then the teacher cannot receive a final summative rating for 2013-14. This scenario does not apply to principals, whose evaluations already include both Administrator Goals and the SGO Average in addition to potential mSGP scores.

11

Evaluation Score Certification Tool

• Districts will have an opportunity to certify that all 2013-14 evaluation data is correct or to make changes where necessary.

– The Department will release the 2013-14 Evaluation Score Certification Tool, a new electronic application for districts to use in certifying final 2013-14 summative ratings for all educators, in late January.

– This interface will allow districts to review data, correct any errors that occurred in the original NJ SMART submission, and certify the accuracy of each staff member’s final score.

– Districts will have approximately one month to complete this process after release of the tool.

12

Next Steps

• Districts: download secure mSGP reports and provide individual educator reports confidentially, ideally through one-on-one conference– Department resources:

http://www.nj.gov/education/AchieveNJ/teacher/percentile.shtml

• Educators: review data, share questions/feedback with supervisor

• Districts Identifying Data Issues: consult Department guidance: http://www.nj.gov/education/AchieveNJ/resources/1314mSGPdataprocedures.pdf

• Department: release 2013-14 Evaluation Score Certification Tool in coming weeks

• Districts: certify all 2013-14 evaluation scores within 4 weeks of Tool access

13

Implementation Timeline: Common Core, State Assessments, and Student Growth Data

CCSS curriculu

m alignment begins

(K-2 math)

All curriculum aligned to

CCSS

NJ ASK aligned to

CCSS in ELA (3-8) and Math (3-5)

2011-12 median Student Growth

Percentiles (mSGPs)

released to pilot

districts

NJ ASK completely aligned to

CCSS

PARCC piloted in

classrooms across 1,276

schools

2012-13 mSGPs

released to all districts as practice

exercise

Full PARCC Implemen

tation

2013-14 mSGP data

released

‘10-’11

‘12-’13

‘13-’14

‘14-’15

‘11-’12

Setting the Context

14

SGP and the PARCC Transition

Multi-Year Preparation•CCSS adopted 4 years ago •NJ ASK incrementally aligned to CCSS in content and rigor•2014 NJ ASK scores show little change compared to prior years despite increased rigor•81% of schools who will use PARCC engaged in pilot testing technology

Growth, Not Proficiency

•To calculate SGP, student growth is compared with growth of academic peers taking the same assessments statewide•SGP does not depend on tests having a consistent scale and is not a criterion-reference metric

Multiple Measures,

Lower Stakes•mSGP is one of multiple measures for the educators in tested grades and subjects; others include educator practice, SGOs, and additional goals and evaluation leadership for principals•mSGP weight reduced to 10% for all educators for 2014-15 to recognize adjustment to new assessment

15

Appendix: Calculating SGP and mSGP

• The following slides depict and explain how SGP and mSGP scores are calculated.

• For a video overview of SGP calculation, please access http://survey.pcgus.com/njgrowth/player.html.

16

Calculating Student Growth Percentiles

• Student Growth Percentiles (SGPs) measure how much a student has learned from one year to the next compared to students with a similar NJ ASK performance history from across the state (“academic peers”).

• SGP scores range from 1 to 99 All students can show growth.

17

Why Student Growth?

A student’s NJ ASK score does not tell the whole story.

Partially Proficient

Gr. 3 Gr. 4 Gr. 5

100

200

250

150160 165

NJ ASK Scale Score by Grade

NJ A

SK

Sca

le

Proficient

Advanced Proficient

230205

220

Albert

Maria

Under our current system, schools and parents might only notice that Maria is “Proficient” and that Albert is “Partially Proficient.”

18

SGP Considers Growth, Not Proficiency

Albert’s Prior

Scores

Academic Peers’ Prior

Scores

Albert has taken the 5th-grade NJ ASK. How does his score compare to those of

his academic peers?

3rd Gr. 150

4th Gr. 160

5th Gr. 165

3rd Gr. ≈150

4th Gr. ≈160

5th Gr. ???

19

Understanding Academic Peers

• Multiple consecutive years of data are required to calculate SGP.– Since 3rd grade is the first year of the NJ ASK, 4th grade is the

first year for which we can calculate SGP.

• The more years of data available, the more precise an academic peer group is.– By 8th grade, we have up to 6 years of scores for the calculation

(as long as students have taken the same assessments).– If a student does not have an NJ ASK score for a given year, an

SGP cannot be calculated until he/she has 2 consecutive years of scores.

• The peer group includes all students in a grade cohort who have taken the exact same assessments, i.e., all 7th graders who took the 3rd, 4th, 5th and 6th-grade Math NJ ASK and the 7th-grade Math NJ ASK

• Academic peer groups are recalculated every year, considering all assessment data for a given cohort of students.

20

Determining an SGPN

J A

SK

Sco

re

Partially Proficient100

200

150160 165

Proficient

Advanced Proficient

250

Gr.3 Gr.5 Gr.4

Partially Proficient100

200

150160

200Proficient

Advanced Proficient

110

250

Gr.3 Gr.5 Gr.4

Albert’s 5th-Grade NJ ASK ScoreAlbert’s Academic Peers’ NJ ASK Scores

Albert scored 165. His academic peers scored between 110 and 200. How did Albert do in

comparison to them?

29%

70%

165

21

Determining an SGP

A comparison to his academic peers allows us to see that Albert actually outperformed 70% of students who, up

until this year, performed in a similar manner to Albert. Albert’s SGP score would be 70.

1%

99%

70%

Albert’s Score

Academic Peers’ Scores

5th Gr. 165

5th Gr.

110 - 200

SGP

22

Determining a Teacher’s mSGP Rating

22

Albert’s teacher would receive a

median SGP score of 51.

Albert’s SGP is

arranged along with the SGPs of

all his teacher’s students

from low to high.

Student SGP ScoreHugh 12Eve 16Clarence 22Clayton 24Earnestine 25Helen 31Clinton 35Tim 39Jennifer 44Jaquelyn 46Lance 51Roxie 53Laura 57Julio 61Selena 65Ashlee 66Albert 70Mathew 72Marcus 85Charles 89Milton 97

23

mSGP Technical Rules

In future years, teachers should know that:

• If two or three years of data are available, the Department will use the best available score — either the teacher’s median score of their current roster or the median of all student scores over the years available.

• The mSGP score, along with the observation and SGO scores, will be placed on a 1.0 - 4.0 scale and weighted appropriately to determine a summative evaluation rating.

Student SGP Score

Hugh 12Eve 16Clarence 22Clayton 24Earnestine 25Helen 31Clinton 35Tim 39Jennifer 44Jaquelyn 46Lance 51Roxie 53Laura 57Julio 61Selena 65Ashlee 66Albert 70Mathew 72Marcus 85Charles 89Milton 97

Mel 56Laura 57Regina 58Marissa 60Julio 61Faye 63Selena 65Ashlee 66Jackie 67Courtney 68Albert 70Matthew 72Laura 77Jack 78Jared 80Rick 84Marcus 85Charles 89Michelle 92Molly 95Milton 97

Student SGP Score

Hugh 12Eve 16John 16Charles 20Annie 20Clarence 22Clayton 24Earnestine 25Jake 27Helen 31Rachel 33Clinton 35Tim 39George 41Amber 42Jennifer 44Jaquelyn 46Bobby 50Lance 51Roxie 53Mike 55

Median over 2 years = 56Median this Year = 51

24

mSGP Conversion from 1 - 99 to 1.0 - 4.0

mSGP Score

Evaluation Rating

1 – 20 121 1.122 1.223 1.324 1.425 1.526 1.627 1.728 1.829 1.930 231 2.132 2.233 2.334 2.4

mSGP Score

Evaluation Rating

65 3.566 3.567 3.568 3.669 3.670 3.671 3.772 3.773 3.774 3.875 3.876 3.877 3.978 3.979 3.9

80 - 99 4

mSGP Score

Evaluation Rating

35 2.536 2.537 2.638 2.639 2.740 2.741 2.842 2.843 2.944 2.945 346 347 348 349 3

mSGP Score

Evaluation Rating

50 351 352 353 354 355 356 3.157 3.158 3.259 3.260 3.361 3.362 3.463 3.464 3.4

Based on the mSGP score, Albert’s teacher receives a rating of 3 for this component, which is then combined with other

evaluation components to get a summative rating.

25

SGP Conversion Chart Explained

mSGP ScoreEvaluation

Rating35 2.536 2.537 2.638 2.639 2.740 2.741 2.842 2.843 2.944 2.945 3.046 3.047 3.048 3.049 3.050 3.051 3.052 3.053 3.054 3.055 3.056 3.157 3.158 3.259 3.260 3.361 3.362 3.463 3.464 3.4

Why are all the values between 45 and 55 set to the same score (3.0)? • The Department believes that

educators in the middle of the mSGP distribution are driving significant academic growth in their students.

• Educators whose students achieve scores in this range should be recognized by receiving a rating on par with their impact.

26

SGP Conversion Chart Explained

mSGP Score

Evaluation Rating

1 – 20 1.0

21 1.122 1.223 1.324 1.425 1.526 1.627 1.728 1.829 1.930 2.031 2.132 2.233 2.334 2.4

Why are the values at the extreme ends of the distribution, 1-20 = 1 in this case (and 80-99 = 4), set to the same score? • When more than half of a

teacher's students are in the top 20 percentile points on the SGP scale it is an indication of very high growth.

• When more than half of a teacher's students are in the bottom percentile points this is an indicator of low growth to be considered with other evidence.

mSGP Score

Evaluation Rating

65 3.566 3.567 3.568 3.669 3.670 3.671 3.772 3.773 3.774 3.875 3.876 3.877 3.978 3.979 3.9

80 - 99 4.0

27

SGP Conversion Chart Explained

mSGP Score

Evaluation Rating

65 3.566 3.567 3.568 3.669 3.670 3.671 3.772 3.773 3.774 3.875 3.876 3.877 3.978 3.979 3.9

80 - 99 4.0

Why Decimals? Why Tenths? • The use of decimals instead of whole

numbers enables the scale to increase/decrease gradually, improving the statistical efficiency of the conversion.

• This prevents large rating differences that may not accurately reflect significant differences in student learning.