Http://miblsi.cenmi.org Using CBM to Predict MEAP Reading Test Scores.

http://miblsi.cenmi.org

Using CBM to Predict MEAP Reading Test Scores

2

Our Research Team

Anna Harms Patrick SorrelleCristy Coughlin

Ed HuthCheyne LeVesseurChristine Russell

3

1. Curriculum-based measures for reading provide us with tools to predict performance on high-stakes assessments

2. We must know how our screening tools and cut scores are designed to function

3. We must use a data-driven process for selecting screening tools and cut scores

3 Big Ideas to Take Away

4

Background

4

Purpose of MiBLSi

A statewide structure to create capacity for an integrated behavior and academic Multi-Tiered System of Support (MTSS) that can be implemented with fidelity, is sustainable over time and utilizes data-based decision making at all levels of implementation support

Provides guidance, visibility, funding, political support for MiBLSi

Students

Building Staff

Building Leadership Team

LEA District Cabinet and

Implementation Team

Across State

Multiple ISD/LEA Teams

All staff

All students

Multiple schools w/in local district

Provides guidance, visibility, funding, political support, and implementation supports

Provides coaching and TA for LEA and/or ISD Teams

Provides guidance and manages implementation

Provides effective practices to support students

Improved behavior and reading

ISD Cabinet and Implementation

Team

Regional Technical

Assistance

MichiganDepartment of

Education/MiBLSi Leadership

Multiple LEAs w/in intermediate district

Provides guidance, visibility, funding, political support, and implementation supports

MiBLSi Statewide Structure of Support

Who is supported?

How is support provided?

7

2014

New ISDs

Previous ISDs

Check-In Partnership ISDs

8

2014

Partnering Districts

9

• A need to accurately respond to requests for recommendations on how to select an appropriate set of benchmark goals and cut points for risk

• DIBELS Next, Introduction of the Composite Score

• AIMSweb Default Criteria• University of Oregon, Release of an Alternate

Set of Benchmark Goals and Cut Points for Risk

• A desire to “see for ourselves” how well different benchmark goals and cut points for risk function with the MEAP reading test

What Led to This Study

10

• A desire to become better consumers of research by learning how to conduct research

• We did NOT set out to create a new set of benchmark goals and cut points for risk

What Led to This Study

11

Provided initial information during an MTSS Coordinator webinar late Fall 2013

Further invited ISDs partnering with MiBLSi and held follow-up meetings with ISDs that expressed interest

Took those who volunteered and were able to successfully export, de-identify, and submit data to MiBLSi by early Winter 2014

Recruitment

12

Information on Study SampleNumber of ISDs, Districts and Students

Districts Included in the Study

Students Included in the Study

ISD 1 11 18,180

ISD 2 2 2,456

ISD 3 3 2,312

ISD 4 1 224

Total: 4 17 23,172

13

Information on Study SampleDemographics

RaceState Totals

Study District Totals

White 69.0% 79.0%

African American 18.3% 1.9%

Hispanic 6.4% 13.2%

Asian 2.9% 2.8%

Multi-Racial 2.5% 2.5%

American Indian 0.8% 0.5%

Native Hawaiian 0.1% 0.1%

Socioeconomic Status

State Totals

Study District Totals

Eligible Free and Reduced Lunch

48.1% 33.8%

14

Information on Study Sample2012 MEAP Reading Results

Grade Level

State-wide Average Meeting

Proficient or Above

Study DistrictsAverage Meeting

Proficient or Above

3rd Grade 66% 77%

4th Grade 68% 80%

5th Grade 70% 81%

6th Grade 68% 77%

7th Grade 62% 73%

8th Grade 66% 77%

15

Information on Study SampleSample Size Ranges by Grade

DIBELS NextFall, Winter and Spring

2011 - 2012Fall 2012

2nd Grade 1006 - 1624

3rd Grade 1342 - 1570 1006 - 2604

4th Grade 1483 - 1609 1342 - 2446

5th Grade 1451 - 1662 1483 - 2297

6th Grade 519 - 528 365 - 592

AIMSwebFall, Winter and Spring

2011 - 2012Fall 2012

6th Grade 413 - 417 441 - 456

7th Grade 687 - 708 806 - 808

8th Grade 732

16

This presentation will not provide detailed background information about:

• MTSS• Universal Screening• DIBELS Next• AIMSweb• MEAP

We are assuming that you have familiarity with these topics or have a way to deepen your knowledge outside of this presentation.

Assumptions

17

Reading CBM Correlation with High-Stakes Assessments

18

Assessments Used Within the Study

DIBELS NextFall 2011

Winter 2012

Spring 2012

Fall 2012

Composite 2nd - 6th 2nd - 6th 2nd - 6th 3rd-6th

DORF Words Correct 2nd - 6th 2nd - 6th 2nd - 6th 3rd-6th

DORF Accuracy 2nd - 6th 2nd - 6th 2nd - 6th 3rd-6th

Retell 2nd - 6th 2nd - 6th 2nd - 6th 3rd-6th

Daze 3rd-6th 3rd-6th 3rd-6th 3rd-6th

NWF WWR 2nd

NWF CLS 2nd

2012 MEAP Reading 3rd-6th

AIMSwebFall 2011

Winter 2012

Spring 2012

Fall 2012

6th - 7th 6th - 7th 6th - 7th 6th - 8th

R-CBM X X X X

MAZE X X X X

2012 MEAP Reading X

19

• MEAP is a standards-based test that measures how well Michigan students in 3rd through 9th grade are achieving

• Includes tests in reading, writing, math, science and social studies

• Reward, Focus and Priority schools are chosen based on MEAP results

• MEAP is the measuring stick for “Adequate Yearly Progress”

• Parents utilize MEAP results as a standard for school success

In Michigan MEAP is Important

20

2012 MEAP Reading Results

Grade Level

State-wide Average Meeting

Proficient or Above

3rd Grade 66%

4th Grade 68%

5th Grade 70%

6th Grade 68%

7th Grade 62%

8th Grade 66%

21

What if we didn’t have to wait and see how students do on the MEAP?

What if we could quickly

and accurately predict a student’s

future reading performance?

22

Good news, we have universal screening tools to help us!

Rea

ding

gra

de le

vel

Grade level corresponding to age 1 2 3 4

4

3

2

1

5

2.5

5.2

Early Intervention Changes Reading Outcomes

At Risk on Early Screening

Low Risk on Early Screening

3.2

Control

With research-based core but without extra instructional intervention

4.9

Interventio

n

With substantial instructional intervention

Torgesen, 2008

24

• Systematic assessment of all students within a school or district, on academic and/or social-emotional indicators for the purpose of identifying system-wide needs as well as intervention supports

• Provides information for problem solving

at multiple educational levels

• Identify those who are at-risk for poor

performance on other critical measures• Later reading assessments• High-stakes assessment / MEAP

Why Universal Screening?

Students

Building Staff

Building Leadership Team

LEA District Cabinet and

Implementation Team

25

CBM Correlations with

Fall 2012 MEAP Results

Universal Screening Concurrent

Fall 2012

MEAPFall 2012

Universal Screening Previous Spring 2012

Universal Screening Previous Winter 2012

Universal Screening Previous

Fall 2011

26

• A correlation tells us how two things relate

• Does not tell us what causes the outcomes

• When two variables correlate well, we can predict results on one based on the other

Why Would a Correlation Be Helpful?

27

When is a correlation meaningful?

Expected Correlations Coefficient

Very LargeMeasuring the same thing possibly in a different format

.70 - .99

LargeMeasuring the same construct but through different assessments

.5 - .69

ModerateMeasuring similar things .30 - .49

Small/TrivialMeasuring unrelated things .00 - .29

Hopkins, 2002

28

CBM Correlations with

Fall 2012 MEAP Results

Universal Screening Concurrent

Fall 2012

MEAPFall 2012

Universal Screening Previous Spring 2012

Universal Screening Previous Winter 2012

Universal Screening Previous

Fall 2011

ConcurrentAssessments

29

CBM to MEAP Correlations Concurrent Fall 2012

Third Grade Fourth Grade Fifth Grade Sixth Grade

Composite .65 .64 .65 .53

DORF WC .60 .59 .60 .49

DORF Accuracy

.51 .45 .45 .33

DAZE .53 .59 .61 .47

Large =

Moderate =

DIBELS Next

AIMSwebSixth Grade Seventh Grade Eighth Grade

R-CBM .54 .55 .55

MAZE .49 .48 .56

30

Higher Correlation = More Variance Explained

31

Value of the DIBELS Next Composite

DORF Score Predicting

MEAP

DIBELS Composite Score Predicting

MEAP

3rd Grade .60 .65

4th Grade .59 .64

5th Grade .60 .65

6th Grade .49 .53

Additional Variance Explained by

DIBELS Composite Score

7.36%

6.41%

6.50%

3.56%

32

DIBELS Next Correlations with

4th Grade Fall 2012 MEAP Results

Universal Screening 4th Grade

Fall 2012

MEAP4th Grade Fall 2012

Universal Screening 3rd Grade

Spring 2012


Winter 2012


Fall 2011

33

DIBELS Next Correlations with

4th Grade Fall 2012 MEAP Results

Previous Fall 2011

Previous Winter 2011

Previous Spring 2011

Concurrent Fall 2012

Composite .63 .60 .63 .64

DORF WC .59 .60 .60 .59

DORF Accuracy

.47 .39 .46 .45

DAZE .50 .52 .46 .59

Large =

Moderate =

34

When a classroom of students are screened in

reading

Some students will be on track

Some students will be at risk

We’ve CBMed…Now What?

Some students will be at significant risk

35

• At Risk for later low reading performance

• How do we know if this prediction is accurate?

• AUC• Sensitivity• Odds / Conditional Percentages

At Risk For…

36

• The probability a screening test will correctly classify a pair of students from two different categories (at-risk, not at-risk)

• A measure of a test’s overall diagnostic accuracy (e.g., rapid strep throat test)

Area Under the Curve (AUC)

37

AUCs Range From .5 to 1.0

38

Range across all assessment times within the study:

AUC RangeDiagnostic accuracy of DIBELS Next and

AIMSweb to MEAP

DIBELS Next AIMSweb

2nd Grade .81 - .85 6th Grade .78 - .79

3rd Grade .82 - .85 7th Grade .76 - .84

4th Grade .79 - .83 8th Grade .83

5th Grade .80 - .81

6th Grade .76 - .77

39

• Predict later performance on high-stakes reading assessments

• Nearly all correlations between CBM Reading and MEAP fall in the moderate to large categories

• Majority of AUC values are at or above .80

• DIBELS Next and aimsweb are efficient assessments that allow us to predict with some accuracy up to a year in advance

CBM Reading Assessments

40

We must know how our

screening tools and cut scores

are designed to function

41

Benchmark Goals and Cut Points for Risk Influence Our Decisions

Example A:Fall ORF

benchmark=70 wcpm

Example B:Fall ORF

benchmark = 97 wcpm

42

1. We cannot let any student who might be at risk slip through the cracks

2. We want an accurate picture without overwhelming our resources to provide intervention

Methods for Establishing Benchmark Goals and Cut Points for Risk

43

Classification Accuracy

Sensi

tivity

Specifi

city

44

If we don’t want to miss any students who might be at risk…

…then we may favor the sensitivity statistic when setting goals and cut points for risk

Sensitivity: Proportion of students correctly classified at-risk / nonproficient on both the screening and another critical assessment

Sensitivity

45

Decision rule for establishing the benchmark goal:

• Sensitivity of .90 or higher

Example: University of Oregon

Center on Teaching and Learning Cutpoints

46

GR2 Fall

GR3 Fall

GR4 Fall

GR5 Fall

GR6 Fall

GR2 W

inter

GR3 W

inter

GR4 W

inter

GR5 W

inter

GR6 W

inter

GR2 Spr

ing

GR3 Spr

ing

GR4 Spr

ing

GR5 Spr

ing

GR6 Spr

ing

GR3 Con

curre

nt

GR4 Con

curre

nt

GR5 Con

curre

nt

GR6 Con

curre

nt

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Sensitivity Values of DIBELS Oral Reading Fluency Cut-points in Predicting MEAP Pass/Fail

CTL Cut-point Sensitivity

Sen

siti

vity

Val

ue

47

GR2 Fall

GR3 Fall

GR4 Fall

GR5 Fall

GR6 Fall

GR2 W

inter

GR3 W

inter

GR4 W

inter

GR5 W

inter

GR6 W

inter

GR2 Spr

ing

GR3 Spr

ing

GR4 Spr

ing

GR5 Spr

ing

GR6 Spr

ing

GR3 Con

curre

nt

GR4 Con

curre

nt

GR5 Con

curre

nt

GR6 Con

curre

nt

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1



Sen

siti

vity

Val

ue

SENSITIVITY = 0.90

48

CTL True Positive(Sensitivity)

False Positive

Identified At Risk(Positive)

False NegativeTrue Negative(Specificity)

Identified On Track(Negative)

49

• Quickly identify students that are likely to need additional support to prevent later academic difficulty.

DMG Development of Benchmark Goals and Cut Points for Risk:

50

Did not focus on one single rule or standard.

Primary consideration: • Odds a student has of meeting a future

reading goal based on their DIBELS Next score.

Secondary consideration: • Marginal percents and logistic regression.


51


Cut Point

Odds of Meeting Future Reading Goals and Passing High-Stakes Assessments

At or Above Benchmark: 80-90%

Below Benchmark: 40-60%

Well Below Benchmark: 10-20%

52

Grade 2 Grade 3 Grade 4 Grade 5 Grade 60

20

40

60

80

100

89 8991 91

82

65

59

73

61

47

38

31

47

29

21

Odd of Passing MEAP: DMG Composite Fall 2011

53

Grade 7 Grade 80

20

40

60

80

100

89 90

64

55

41

20

Benchmark

Strategic

Intensive

Odds of Passing MEAP: AIMSweb R-CBM

Fall 2011 to MEAP Fall 2012

54

• CTL Benchmark Goals emphasize sensitivity

• DMG Benchmark Goals emphasize odds / conditional percentages

• What happens when we look at the sensitivity of the DMG goals or the odds of the CTL goals?

Consider…

55

GR2 Fall

GR3 Fall

GR4 Fall

GR5 Fall

GR6 Fall

GR2 W

inter

GR3 W

inter

GR4 W

inter

GR5 W

inter

GR6 W

inter

GR2 Spr

ing

GR3 Spr

ing

GR4 Spr

ing

GR5 Spr

ing

GR6 Spr

ing

GR3 Con

curre

nt

GR4 Con

curre

nt

GR5 Con

curre

nt

GR6 Con

curre

nt

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1



Sen

siti

vity

Val

ue

SENSITIVITY = 0.90

56

GR2 Fall

GR3 Fall

GR4 Fall

GR5 Fall

GR6 Fall

GR2 W

inter

GR3 W

inter

GR4 W

inter

GR5 W

inter

GR6 W

inter

GR2 Spr

ing

GR3 Spr

ing

GR4 Spr

ing

GR5 Spr

ing

GR6 Spr

ing

GR3 Con

curre

nt

GR4 Con

curre

nt

GR5 Con

curre

nt

GR6 Con

curre

nt

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0

50

100

150

200

250

80

97

111 13

2 150

100 11

5 130 15

0

151

111 12

3 144 15

5

162

97

111 13

2 150


CTL Cut-point Value CTL Cut-point Sensitivity

Sen

siti

vity

Val

ue C

ut-p

oin

t Valu

e

SENSITIVITY = 0.90

57

GR2 Fall

GR3 Fall

GR4 Fall

GR5 Fall

GR6 Fall

GR2 W

inter

GR3 W

inter

GR4 W

inter

GR5 W

inter

GR6 W

inter

GR2 Spr

ing

GR3 Spr

ing

GR4 Spr

ing

GR5 Spr

ing

GR6 Spr

ing

GR3 Con

curre

nt

GR4 Con

curre

nt

GR5 Con

curre

nt

GR6 Con

curre

nt

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0

50

100

150

200

250

52

70

90

111

107

72

86

103 12

0

109

87

100 11

5 130

120

70

90

111

107

80

97

111 13

2 150

100 11

5 130 15

0

151

111 12

3 144 15

5

162

97

111 13

2 150


DMG Cutpoint Value CTL Cut-point Value


Sen

siti

vity

Val

ue C

ut-p

oin

t Valu

e

SENSITIVITY = 0.90

58

GR2 Fall

GR3 Fall

GR4 Fall

GR5 Fall

GR6 Fall

GR2 W

inter

GR3 W

inter

GR4 W

inter

GR5 W

inter

GR6 W

inter

GR2 Spr

ing

GR3 Spr

ing

GR4 Spr

ing

GR5 Spr

ing

GR6 Spr

ing

GR3 Con

curre

nt

GR4 Con

curre

nt

GR5 Con

curre

nt

GR6 Con

curre

nt

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0

50

100

150

200

250

52

70

90

111

107

72

86

103 12

0

109

87

100 11

5 130

120

70

90

111

107

80

97

111 13

2 150

100 11

5 130 15

0

151

111 12

3 144 15

5

162

97

111 13

2 150


DMG Cutpoint Value CTL Cut-point Value

DMG Cutpoint Sensitivity CTL Cut-point Sensitivity

Sen

siti

vity

Val

ue C

ut-p

oin

t Valu

e

SENSITIVITY = 0.90

59

Odds of Passing MEAP: DMG-CTL Benchmark Fall 2011 DORF WC to Fall 2012 MEAP


20

40

60

80

100

9289

9289 84

97 95 96 95 95

DMG

CTL

60

Odd of Passing MEAP: Strategic Words Correct Fall 2011


20

40

60

80

100

72

67

75 75

57

88

83 85

77 79

DMG

CTL

61

Odd of Passing MEAP: Intensive Words Correct Fall 2011


20

40

60

80

100

50

37

51

47

24

60

52

63

54

61

DMGCTL

62

We should look at it from the perspective of districts, schools, and grade level

teachers!

Perhaps…

63

What are the Implications for Decision Making?Example: 3rd Grade Fall 2011

CTL WC DMG WC DMG Composite

64

DIBELS CTL

DIBELS DMG

MEAP

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Same District Fall 2011 3rd Grade

65

Same District Fall 2012 3rd Grade Marginal Percents

DIBELS CTL

DIBELS DMG

MEAP

DE Assessment

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

66

Resources to Make Informed Decisions about Measures

67

http://miblsi.cenmi.org/MiBLSiModel/Evaluation/UniversalScreening.aspx

MiBLSi Universal Screening Webpage

68

• Universal Screening Resources Overview

• FAQ on the Topic of Universal Screening

• Universal Screening Annotated Bibliography

• Considerations for Selecting a Universal Screening Measure

• CCSS: Reading and Math

• Sample of Completed Considerations Document (DIBELS Next, SRSS, easyCBM Math, Discovery Education Assessment, NWEA MAP)

MiBLSi Universal Screening Webpage

69

Considerations for

Selecting/Reviewing a

Universal Screening Measure

70

Document Overview

Section Questions

Need in District 1

General Features 2-9

Fit with Current Initiatives/Priorities 10-12

Evidence/Technical Adequacy 12-16

Readiness for Replication 17-19

Resources & Supports 20-32

Capacity to Implement 33-34

71

In God we trust, all others must bring data.

- W. Edwards Deming

See for yourself. . .

Http://miblsi.cenmi.org Using CBM to Predict MEAP Reading Test Scores.

Documents

Transcript of Http://miblsi.cenmi.org Using CBM to Predict MEAP Reading Test Scores.