Preparing for New Test Scores Smarter Balanced assessments measure the full range of the Common...

Smarter – Achievement Level Setting

District Test Coordinator Meeting

December 17, 2014

2

Preparing for New Test Scores

Smarter Balanced assessments measure the full range of the Common Core State

Standards. They are designed to let teachers and parents know whether students are

on track to be college- and career-ready by the time they graduate.

Because the new standards set higher expectations for students—and the new

tests are designed to assess student performance against these higher expectations

—our definition of grade-level performance is higher than it used to be.

As a result, it means that fewer students will meet grade-level standards,

especially for the first few years. Results should improve as students have additional

years of instruction aligned to the new standards and become better equipped to

meet the challenges they present.

This does not mean that our students are “doing worse” than they did last year.

Rather, the scores represent a “new baseline” that provides a more accurate

indicator for educators, students, and parents as they work to meet the rigorous

demands of college and career readiness.

3

Achievement Level Timeline (2014)

• States Approved Achievement Level Setting PlanApril 30

• Online PanelOctober 6-18

• In-Person Panel (Dallas)October 13-19

• Cross-Grade Review Committee (Dallas)October 20

• Technical Advisory Committee ReviewOctober 30• States Vote to Approve Achievement Level

RecommendationsNovember 14

• State Adoption of Achievement LevelsFall/Winter

4









5

Online Panel

Crowd sourcing—thousands of participants

Recommendations broken down by subgroup (teachers, higher education, etc.)

The process involved:Online orientationReview of test questionsRecommended a cut score for the threshold of

Performance Level 3

142 Delawareans participated—exceeded SBAC’s target

6

Three Cut Scores Defines Four Performance Levels

7

Reviewing Items in the Ordered ItemBooklet (OIB) for Bookmarking

1. What do you know about a student who responds successfully to this item; that is, what skills must a student have in order to know the correct answer?

2. What makes this item more difficult than preceding items?

This activity helps you acquire the knowledge to make content-based cut score recommendations

8

Selection of Placement of Bookmark

A threshold Level 3 student would have about a 50% chance of success on the item just before the bookmark

A threshold Level 3 student would have less than a 50% likelihood of success on the items from the bookmark on

9









10

In-Person Panel

K-12 educators, higher education, parents, and community advocates brought together

Recommended cut scores for all Performance Levels

500 participants in grade/subject panels

Nominated by states

25 Delawareans participated

Used Bookmark procedure

Three Rounds in the process

11

Reviewing Items in the Ordered Item Booklet (OIB) for Bookmarking

1. What do you know about a student who responds successfully to this item; that is, what skills must a student have in order to know the correct answer?

2. What makes this item more difficult than preceding items?

This activity helps you acquire the knowledge to make content-based cut score recommendations

12

Three Rounds to Determine Three Cut Scores

13

In-Person Panel Agenda

Day 1Morning: Study Common Core State Standards and

Achievement Level DescriptorsAfternoon: Study the Ordered Item Booklet

Day 2Morning: Bookmark training and Round 1

Make bookmark recommendations individuallyAfternoon: Round 2

Discuss differences in Round 1 bookmark placements at each table

Make Round 2 bookmarks individually

14

In-Person Panel Agenda

Day 3Morning: Round 3

Discuss Round 2 bookmark placements for the entire room (all panelists at all tables)

View supporting data based on Round 2 bookmarks (e.g., impact data, online panel results)

Place Round 3 bookmarks individually Review final recommendations Evaluate the process

15









16

Cross-Grade Review Committee

Also referred to as Vertical Articulation 60 panelists selected from In-Person Panel One Delawarean participated Reviewed Online Panel and In-Person Panel results Recommend changes to “smooth” the data:

Motion Second Discussion Vote: 2/3 majority required Aligned cut scores across grades Eliminated scaled score reversals

One Delawarean participated

17

Vertical Articulation

This is an example to illustrate

Vertical Articulation. It is NOT actual data

from Achievement Level Setting.

18









19

Setting Achievement Levels

States Chiefs reviewReviewed final recommendations from cross-grade

Review CommitteeApproval by State Chiefs

State Board review to seek approval

20

Achievement Level Setting Recommendations

Note that Achievement Level Setting Recommendations are contingent upon approval of the State Board of Education on December 18, 2014. Final vote to adopt will take place on January 15, 2015

21

English Language Arts/Literacy: Threshold Scale Scores

3 4 5 6 7 8 9 10 112300

2350

2400

2450

2500

2550

2600

2650

2700

2750

2800

Level 4

Level 3

Level 2

Grade

Th

resh

old

Sca

le S

core


22

ELA/Literacy: Estimated Percentage of Students at Each Achievement Level

3 4 5 6 7 8 110%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

35% 37% 33% 30% 34% 28% 28%

27% 22%23% 29%

28%31% 31%

20% 23% 29%30% 30% 32% 30%

18% 18% 15% 11% 8% 9% 11%

Level 4

Level 3

Level 2

Level 1

Grade

Est

. %

in

Eac

h L

evel

Estimates based on Spring 2014 Smarter Balanced field test conducted in 21 states and US Virgin Islands.


23

Proposed Scale Score Cut Points

ELA/Literacy Scale Score Thresholds (Scale 2000–3000)

Grade Level 1-to-2 Level 2-to-3 Level 3-to-4

3 2367 2432 2490

4 2416 2473 2533

5 2442 2502 2582

6 2457 2531 2618

7 2479 2552 2649

8 2487 2567 2668

11 2493 2583 2682

For Board Action in January 2015


24

Mathematics: Threshold Scores

3 4 5 6 7 8 9 10 112300

2350

2400

2450

2500

2550

2600

2650

2700

2750

2800

Level 4

Level 3

Level 2

Grade

Th

resh

old

Sca

le S

core


25

Mathematics: Estimated Percentage of Students at Each Achievement Level

3 4 5 6 7 8 110%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

32% 27%35% 35% 36% 38% 40%

29% 36%32% 32% 31% 30% 27%

27% 24% 18% 19% 20% 19% 22%

12% 13% 15% 14% 13% 13% 11%

Level 4

Level 3

Level 2

Level 1

Grade

Est

. %

at

Eac

h L

evel

Estimates based on Spring 2014 Smarter Balanced field test conducted in 21 states and US Virgin Islands.


26

Proposed Scale Score Cut Points

Math Scale Score Thresholds (Scale 2000 – 3000)

Grade Level 1-to-2 Level 2-to-3 Level 3-to-4

3 2381 2436 2501

4 2411 2485 2549

5 2455 2528 2579

6 2473 2552 2610

7 2484 2567 2635

8 2504 2586 2653

11 2543 2628 2718

For Board Action in January 2015


27

Preparing for New Test Scores

Smarter Balanced assessments measure the full range of the Common Core State Standards. They are designed to let teachers and parents know whether students are on track to be college- and career-ready by the time they graduate.

Because the new standards set higher expectations for students—and the new tests are designed to assess student performance against these higher expectations—our definition of grade-level performance is higher than it used to be.

As a result, it means that fewer students will meet grade-level standards, especially for the first few years. Results should improve as students have additional years of instruction aligned to the new standards and become better equipped to meet the challenges they present.

This does not mean that our students are “doing worse” than they did last year. Rather, the scores represent a “new baseline” that provides a more accurate indicator for educators, students, and parents as they work to meet the rigorous demands of college and career readiness.

28

Learn More AboutAchievement Level Setting

Video explanation of processhttps://www.youtube.com/watch?v=bW_yGf4BB1E

Interpretation and use of scoreshttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdf

Easy to follow flow charthttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdf

Q&A documenthttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdf

www.DelExcels.org

https://www.youtube.com/watch?v=bW_yGf4BB1E

http://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdf

http://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdf

http://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdf

http://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdf

http://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdf

http://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdf

http://www.delexcels.org/

Preparing for New Test Scores Smarter Balanced assessments measure the full range of the Common...

Documents

Transcript of Preparing for New Test Scores Smarter Balanced assessments measure the full range of the Common...