Copyright ©2006. Battelle for Kids. Understanding & Using Value-Added Analysis.

Copyright ©2006. Battelle for Kids.

Understanding & UsingValue-Added Analysis

Learning Targets

• Develop confidence in accessing, navigating and interpreting value-added information.

• Understand the connections between value-added information and school improvement.

Introductions

• Name

• Role in Building and/or District

• Draw a face on a post it to indicate your feelings about Value Added

The Work of the Day• Morning will be spent learning about

value-added information:– Conceptual understanding of value-added– Overview of the information available in value-

added reports

• Afternoon will be spent:– Navigating through value-added reports– Learning about people and tools that can

support for your learning– Connecting value-added to school

improvement


Getting Started Value-Added Analysis


Presentation Targets

• Gain an introductory understanding of value-added analysis.

• Acquire an awareness of information available in value-added reports.

• Explore how value-added information can inform school improvement.

• Understand the difference between progress and achievement.


Stair-Step Expectations

Grade Three Grade Four Grade Five

Grade 3 Test Grade 4 Test Grade 5 Test

In a perfect world scores on year-end achievement tests would be enough to document growth. Students would start school at the same place, and move together toward proficiency.

Copyright © 2003. Battelle for Kids


Differentiated Reality

Grade Three Grade Four Grade Five

Students begin school at different places and progress at different rates. This complicates accountability. We need more than scores on a single test to tell the story of a school’s effectiveness.

Grade 3 Test Grade 4 Test Grade 5 Test



Achievement measures alone provide a partial picture:School improvement perspective

Birth Kindergarten 4th Grade Test

Achievement measures are snapshots in time. They provide information on what a child has learned since birth.

If we want to improve schools we need a metric that provides reliable feedback on the work that schools produce.

1st 2nd 3rd


A View of Student Progress:Accountability perspective

30

40

50

60

70

80

90

100

3 4 5 6 7 8

Student AStudent BStandard

We must expect progress for all students.

Student A

Student B

Proficient


The question for educators today

Regardless of where students begin, how do we maximize the progress each

makes in any given year?

84% of the American public believe student progress is

the best measure of a school’s effectiveness!



What is the difference between progress and achievement?


Value-Added Analysis in Ohio

Project SOAR

Ohio’s Value-Added

System


• Operated by the Ohio Department of Education.

• Began as a grade 4 pilot in 2006 in all districts and community schools.

• Provides analysis in math & reading, grades 4-8.

• Uses only state achievement test data.

• Uses the mean gain model.

• Expected growth is set by the State Board of Education.

Ohio’s System


• Operated by Battelle for Kids.

• Began in 2002 with 42 school districts.

• Now includes 100+ districts.

• Provides analysis in all core subjects, grades 3-10.

• Uses state and non-state test data.

• Uses the mean predicted model.

• Expected growth is normative (Average Growth).

Project SOAR (Schools’ Online Assessment Reports)


Why EVAAS™?

• Applies the most sophisticated statistical

methodology available to ensure reliability.

• Allows for the use of all student test data.

• Provides valuable diagnostic information.

• Accommodates different types of test data.

• Used statewide in Tennessee for more than 10 years.


Ohio’s Scale Up Plan

R M Sc SSR M Sc SS

Value-Added in the State Accountability System


• If a district or school demonstrates more than expected growth over two consecutive years it moves up one level in the system.

• If a district or school demonstrates less than expected growth over three consecutive years it moves down one level in the system.

• Value-added measures are significant at one standard error.


How is student assessment data used to produce a value-added growth metric?

Value-Added Analysis*Conceptual Overview


What is a Growth Measure?

Growth = Observed Score – Baseline Score


Value-Added Example

4th 5th 6th 7th 8th

A value-added metric measures the average growth that is produced in one subject in a given year.

This growth is then compared to the amount of growth that would be expected for that group of students. Growth above (or below) expectation is the “value-added.”


Tests must:

• Be highly correlated to curricular objectives.

• Have sufficient stretch.

• Have appropriate levels of reliability.

Test Properties


School Effect ≈ Mean Gain – Expected Growth

Used when all tests across grade levels have consistent scales.

Mean Gain Approach


Mean Gain Example

A crude measure of the growth for this group is 10 NCEs.

Growth = Mean Observed – Mean BaselineGrowth = 35.2 - 25.2 = 10.0

Math scaled scores for the same students.Year 1 • Grade 3Student 1 377 = 26 (NCE)Student 2 380 = 27Student 3 360 = 19Student 4 377 = 26Student 5 381 = 28

Mean Baseline = 25.2

Year 2 • Grade 4Student 1 400 = 37 (NCE)Student 2 385 = 30Student 3 395 = 35Student 4 405 = 41Student 5 390 = 33

Mean Observed = 35.2


Mean Gain Approach (Converted to NCEs)

High Progress Positive School Effects

High Achievement High Mean NCE or Mean Scaled Score

School Effect

Observed

Baseline

Sample School Value-Added Report Math


Mean Gain Approach (Converted to NCEs)


Describe, in conceptual terms, the “mean gain” value-added model.

What NewDiagnostic Information DoValue-AddedReportsProvide?


Value-Added Summary Report

Where are the areas of strength and weakness?


Are the curriculum and instruction working for all students?

School Diagnostic Report


What pattern would represent optimum growth?

Performance Diagnostic Report


How does a student’s growth rate compare to other groups of students?

Student Report


With average growth, where will this student likely score in the future?

Student Projection

Student Search


How can Predictions help identify at risk students?


How could this information change the way educators approach intervention?

Student Search Results


In Summary, Value-AddedInformation Shows…

• The students’ progress made in each subject area and grade level.

• The progress students made who enter the year at different achievement levels.

• How students’ progress in one curricular area or program compares to their progress in another.

• Whether individual students are making adequate progress toward state standards.

Without data, all we have are opinions!


What new information do value-added reports provide?


Why is it important to focus school improvement strategies at the individual grade level and/or subject area?

Copyright ©2006. Battelle for Kids. Understanding & Using Value-Added Analysis.

Documents

Transcript of Copyright ©2006. Battelle for Kids. Understanding & Using Value-Added Analysis.