1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s...
-
Upload
eustace-wilkinson -
Category
Documents
-
view
221 -
download
0
description
Transcript of 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s...
![Page 1: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/1.jpg)
11
Maximizing Predictive Accuracy of District Benchmarks
Illuminate Education, Inc.Illuminate Education, Inc.User’s ConferenceUser’s Conference
Aliso Viejo, California
June 4&5, 2012
![Page 2: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/2.jpg)
22
Maximizing Predictive Accuracy of District Benchmarks
Objective1.Better understand how
performance level setting is key to predictive validity.
2.Better understand how to create performance level bands based on equipercentile equating
![Page 3: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/3.jpg)
33
Comparing District Benchmarks to CST Results
Common Methods for Setting Cutoffs on District Benchmarks:
Use default settings on assessment platform (e.g. 20%, 40%, 60%, 80%) Ask curriculum experts for their opinion of where cutoffs should be set Determine percent correct corresponding to performance levels on CSTs and apply to benchmarks
![Page 4: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/4.jpg)
44
Comparing District Benchmarks to CST Results
There is a better way!
![Page 5: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/5.jpg)
55
Comparing District Benchmarks to CST Results
“Two scores, one on form X and the other on form Y, may be considered equivalent if their corresponding percentile ranks in any given group are equal.” (Educational Measurement-Second Edition, p. 563)
![Page 6: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/6.jpg)
66
Comparing District Benchmarks to CST Results
Equipercentile Method of Equating at the Performance Level Cut-points Establishes cutoffs for benchmarks at
equivalent local percentile ranks as cutoffs for CSTs
By applying same local percentile cutoffs to each trimester benchmark, comparisons across trimesters within a grade level are more defensible
![Page 7: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/7.jpg)
77
Equipercentile Equating MethodStep 1-Identify CST SS Cut-points
![Page 8: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/8.jpg)
88
Equipercentile Equating Method
Step 2 - Establish Local Percentiles at CSTPerformance Level Cutoffs (from scaled score frequency distribution)
![Page 9: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/9.jpg)
99
Equipercentile Equating Method
Step 3 – Locate Benchmark Raw ScoresCorresponding to the CST CutoffPercentiles (from benchmark raw scorefrequency distribution)
![Page 10: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/10.jpg)
1010
2nd Semester
BiologyOld Cutoff FBB BB Basic Proficient Advanced Total0-17 FBB 57 72 25 1 0 15518-34 BB 118 297 511 60 4 99035-48 Basic 19 51 427 401 45 94349-62 Proficient 1 5 27 141 207 38163-70 Advanced 0 0 0 0 20 20
Total 195 425 990 603 276 2489
Correct Classification: Proficient & Advanced on CST = 42%Correct Classification: Each Level on CST = 38%
2006 CST
Equipercentile Equating MethodStep 4 – Validate Classification Accuracy – Old Cutoffs
![Page 11: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/11.jpg)
1111
2nd SemesterBiologyOld Cutoff FBB BB Basic Proficient Advanced Total0-17 FBB 57 72 25 1 0 15518-34 BB 118 297 511 60 4 99035-48 Basic 19 51 427 401 45 94349-62 Proficient 1 5 27 141 207 38163-70 Advanced 0 0 0 0 20 20
Total 195 425 990 603 276 2489
Correct Classification: Proficient & Advanced on CST = 42%Correct Classification: Each Level on CST = 38%
2006 CST
Equipercentile Equating MethodStep 4 – Validate Classification Accuracy – Old Cutoffs
![Page 12: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/12.jpg)
1212
2nd SemesterBiologyOld Cutoff FBB BB Basic Proficient Advanced Total0-17 FBB 57 72 25 1 0 15518-34 BB 118 297 511 60 4 99035-48 Basic 19 51 427 401 45 94349-62 Proficient 1 5 27 141 207 38163-70 Advanced 0 0 0 0 20 20
Total 195 425 990 603 276 2489
Correct Classification: Proficient & Advanced on CST = 42%Correct Classification: Each Level on CST = 38%
2006 CST
Equipercentile Equating MethodStep 4 – Validate Classification Accuracy – Old Cutoffs
![Page 13: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/13.jpg)
1313
2nd SemesterBiologyNew Cutoff FBB BB Basic Proficient Advanced Total0-19 FBB 89 107 53 4 0 25320-26 BB 59 142 148 12 0 36127-40 Basic 39 161 596 176 9 98141-51 Proficient 8 12 181 354 82 63752-70 Advanced 0 3 12 57 185 257
Total 195 425 990 603 276 2489
Correct Classification: Proficient & Advanced on CST = 77%Correct Classification: Each Level on CST = 55%
2006 CST
Equipercentile Equating MethodStep 4 – Validate Classification Accuracy –
New Cutoffs
![Page 14: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/14.jpg)
1414
2nd SemesterBiologyNew Cutoff FBB BB Basic Proficient Advanced Total0-19 FBB 89 107 53 4 0 25320-26 BB 59 142 148 12 0 36127-40 Basic 39 161 596 176 9 98141-51 Proficient 8 12 181 354 82 63752-70 Advanced 0 3 12 57 185 257
Total 195 425 990 603 276 2489
Correct Classification: Proficient & Advanced on CST = 77%Correct Classification: Each Level on CST = 55%
2006 CST
Equipercentile Equating MethodStep 4 – Validate Classification Accuracy –
New Cutoffs
![Page 15: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/15.jpg)
1515
BiologyNew Cutoff FBB BB Basic Proficient Advanced Total0-19 FBB 89 107 53 4 0 25320-26 BB 59 142 148 12 0 36127-40 Basic 39 161 596 176 9 98141-51 Proficient 8 12 181 354 82 63752-70 Advanced 0 3 12 57 185 257
Total 195 425 990 603 276 2489
Correct Classification: Proficient & Advanced on CST = 77%Correct Classification: Each Level on CST = 55%
2006 CST
Equipercentile Equating MethodStep 4 – Validate Classification Accuracy –
New Cutoffs
![Page 16: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/16.jpg)
1616
Example: Classification AccuracyBiology
Old New
2nd SemesterProficient or Advanced 42% 77%Each Level 38% 55%
1st SemesterProficient or Advanced 30% 77%Each Level 31% 50%
![Page 17: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/17.jpg)
1717
Example: Classification AccuracyBiology
Old New
1st QuarterProficient or Advanced 53% 71%Each Level 41% 46%
![Page 18: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/18.jpg)
1818
Example: Classification AccuracyChemistry
Old New2nd Semester: Prof. & Adv. 63% 79%2nd Semester: Each Level 47% 52%1st Semester: Prof. & Adv. 74% 74%1st Semester: Each Level 49% 50%1st Quarter: Prof. & Adv. 83% 76%1st Quarter: Each Level 48% 47%
![Page 19: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/19.jpg)
1919
Example: Classification AccuracyEarth Science
Old New
2nd Semester: Prof. & Adv. 48% 68%2nd Semester: Each Level 43% 52%1st Semester: Prof. & Adv. 33% 66%1st Semester: Each Level 38% 47%1st Quarter: Prof. & Adv. 42% 56%1st Quarter: Each Level 34% 41%
![Page 20: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/20.jpg)
2020
Example: Classification AccuracyPhysics
Old New2nd Semester: Prof. & Adv. 57% 87%2nd Semester: Each Level 37% 57%1st Semester: Prof. & Adv. 60% 88%1st Semester: Each Level 42% 50%1st Quarter: Prof. & Adv. 65% 87%1st Quarter: Each Level 47% 45%
![Page 21: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/21.jpg)
2121
Things to Consider Prior to Establishing the Benchmark Cutoffs
Will there be changes to the benchmarks after CST percentile cutoffs are established? If NO then raw score benchmark cutoffs can be
established by linking CST to same year benchmark administration (i.e. spring 2011 CST matched to 2010-11 benchmark raw scores)
If YES then wait until new benchmark is administered and then establish raw score cutoffs on benchmark
How many cases are available for establishing the CST percentiles? (too few cases could lead to unstable percentile distributions)
![Page 22: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/22.jpg)
2222
Things to Consider Prior to Establishing the Benchmark Cutoffs (Continued)
How many items comprise the benchmarks to be equated? (as test gets shorter it becomes more difficult to match the percentile cutpoints established on the CST’s)
![Page 23: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/23.jpg)
2323
SummaryEquipercentile Equating Method
Method generally establishes a closer correspondence between the CST and Benchmarks
Comparisons between benchmark and CST performance can be made more confidently
Comparisons between benchmarks within the school year can be made more confidently
![Page 24: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/24.jpg)
2424
Coming Soon from Illuminate Education, Inc.!
Reports using the equipercentile methodology are being programmed to:
(1) establish benchmark cutoffs for performance bands
(2) create validation tables showing improved classification accuracy based on the method
![Page 25: 1 Maximizing Predictive Accuracy of District Benchmarks Illuminate Education, Inc. User’s Conference Aliso Viejo, California June 4&5, 2012.](https://reader034.fdocuments.net/reader034/viewer/2022051301/5a4d1b1a7f8b9ab059993169/html5/thumbnails/25.jpg)
Contact:Tom Barrett, Ph.D.President, Barrett Enterprises, LLC951-905-5367 (office)951-237-9452 (cell)[email protected]
2525