Do Schools Mater for High Math Achievement? Evidence from ...
Raising the Bar for Oregon. Adopt New Math Cut Scores and Final Math Achievement Level Descriptors...
28
Oregon’s Math Achievement Standards Raising the Bar for Oregon
Transcript of Raising the Bar for Oregon. Adopt New Math Cut Scores and Final Math Achievement Level Descriptors...
Oregon’s Math Achievement Standards Raising the Bar for Oregon
Adopt New Math Cut Scores and Final Math
Achievement Level Descriptors and Policy
Definitions
Adopt High School Math Achievement Level
Descriptors & Review 3-3 Math Achievement Level
Descriptors
Adopt Grades 3-8 Math Achievement Level Descriptors
New Mathematics Content Standards were adopted for grades K-8 in 2007 and high school in 2009.
Federal law requires that Achievement Standards align with Content Standards.
Why Now?
Alignment Across Grades – Earlier grades should better predict performance in later grades
Students and parents need better information about the level of skill needed to succeed in High School
Decision today is essential
Why Now?
District mathematics experts created the achievement level descriptors based on their in-depth understanding of the mathematics content standards
District and School mathematics experts as well as representatives from the community recommended a cut score based on their understanding of the content standards, achievement level descriptors, ODE’s projected cut-scores and a review of the Impact data
Process
Current Cut-ScoresGrade Nearly
Meets Meets Exceeds
3 201 205 217
4 208 212 225
5 214 218 229
6 216 221 232
7 221 226 238
8 225 230 241
HS 231 236 246
*Panel Recommended 237 for HS
Staff Recommended ScoresRecommend Cut Scores
Impact Data
Grade Nearly Meets
Meets
Exceeds Projected % Meets & Exceeds
2009-10 % Meets & Exceeds (Actual)
3 205 212 219 47% 79%4 212 219 227 46% 79%5 219 225 234 45% 79%6 222 227 237 46% 74%7 228 232 242 51% 80%8 230 234 245 53% 72%
HS 232 236* 251 56% 56%*Panel Recommended 237 for HS
Districts and Schools have received repeated financial hits, and do not have additional funds for professional development
It will be easy to misinterpret the data and believe that student performance declined
AYP targets increase to 70% this year
This will create a discrepancy with Reading that will be hard for Districts to explain
Concerns Raised
There are built-in transitions for status and “safe harbor”
ODE plans to work with stakeholders to identify strategies to address these new complexities. We will examine state and federal accountability in light of these changes
Impact of Achievement Standards on AYP
Agree Disagree No Opinion
Should Oregon have a goal of establishing achievement standards that are similar to other higher performing states such as Washington and Minnesota?
69.4% 25.6% 5.1%
Are the recommended mathematics achievement standards reasonable and appropriate?
For Grades 3 and 4 37.2% 46.5% 16.2%
For Grades 5 and 6 36.0% 48.6% 15.4%
For Grades 7 and 8 32.9% 41.3% 25.8%
For High School 37.2% 34.9% 27.8%
Public Opinion Survey Results
79
69
86
6266
8183
71
59
5147
36
61
28 30
51 52
35
24 25
0
10
20
30
40
50
60
70
80
90
100
Grade 3: Current Cut Score = 205 Recommended Cut Score = 212
Current Cut Scores
Recommended Cut Scores
79
67
85
61
66
7983
71
58
5046
32
61
28 29
4851
34
2023
0
10
20
30
40
50
60
70
80
90
100
Grade 4: Current Cut Score = 212 Recommended Cut Score = 219
Current Cut Scores
Recommended Cut Scores
79
68
86
60
68
83 82
71
55
4845
29
60
2629
5350
33
1720
0
10
20
30
40
50
60
70
80
90
100
Grade 5: Current Cut Score = 218 Recommended Cut Score = 225
Current Cut Scores
Recommended Cut Scores
74
61
82
56
60
79 79
65
40 39
46
32
63
28 30
52 51
34
1317
0
10
20
30
40
50
60
70
80
90
100
Grade 6: Current Cut Score = 221 Recommended Cut Score = 227
Current
Recommended
80
69
90
61
69
83 83
72
52
44
51
34
69
3034
58 56
38
16 18
0
10
20
30
40
50
60
70
80
90
100
Grade 7: Current Cut Score = 226 Recommended Cut Score = 232
Current
Recommended
72
60
83
51
57
7476
62
35 33
53
40
72
34 36
5458
40
18 19
0
10
20
30
40
50
60
70
80
90
100
Grade 8: Current Cut Score = 230 Recommended Cut Score = 234
Current
Recommended
56
38
72
31
38
5861
42
17 17
56
38
72
31
38
5861
42
17 17
0
10
20
30
40
50
60
70
80
90
100
Grade 10: Current Cut Score = 236 Recommended Cut Score = 236
Current
Recommended
We are 3 years into the implementation of Oregon’s Math Content Standards. The Common Core standards were finalized in 2010.
Raising our Achievement Standards and expecting more of our students now will help us transition to the higher standards of the Common Core.
These new Achievement Standards bring Oregon’s standards closer to those of other states including Washington.
How Does this Relate to the Common Core?
Staff Recommended Extended Assessment Scores
Recommend Cut Scores
Impact Data
Grade Nearly Meets
Meets
Exceeds Projected % Meets & Exceeds
2009-10 % Meets & Exceeds (Actual)
3 97 104 112 26% 62%4 98 106 114 26% 57%5 101 110 118 25% 57%6 99 103 110 19% 56%7 101 102 107 25% 46%8 101 105 110 18% 47%
HS 99 106 115 13% 50%
Oregon Department of Education Standards Verification
Mathematics, grades 3-8 and High School
August 2010
20
Presented to the Oregon State Board of Education
David T. Conley, PhDMary Seburn, PhD
Liz Gilkey, JD
Standards Verification Process
• Bookmarking Method• Modified to include comparative data
• Allowed panelists to compare Oregon’s current and proposed cut scores to other states and countries• Included external data from NAEP, PISA, other states• Included projected (derived) scores based on higher standards and
increased predictive power
• 42 Oregon stakeholders recommended cut scores over 3 days
• Panelists recommended raising achievement standards for all grade levels
• External evaluators monitored process and documented observed evidence of validity
21
22
Evaluation• External Evaluation conducted by The Educational Policy
Improvement Center (EPIC)• Observed process• Conducted formal and informal Interviews with participants• Conducted training and workshop evaluations
• Documented implementation of best practices and technical adequacy• Noted evidence of procedural validity as observed• Noted when standards for Educational and Psychological Testing
were met• Documented technical evidence required by NCLB Peer Review• Documented adherence to best and emerging practice
23
Training Evaluation Overall, feedback on the training was positive, for
example:• 88.1% of participants agreed or strongly agreed with the
statement, “The training materials were helpful.”
• 92.9% of participants agreed or strongly agreed with the statement, “I am confident I understand my role in the standards verification process.”
• 71.4% of participants agreed or strongly agreed with the statement, “Overall, I feel well trained and prepared to complete the standards verification task.”
24
Workshop Evaluation Overall, feedback on the process was positive, for
example:• 81.4% of participants agreed or strongly agreed with the
statement, “The Bookmark Procedure was well described.” • 97.6% of participants agreed or strongly agreed with the
statement, “Overall, I am satisfied with my group's final bookmarks.”
• 83.7% of participants agreed or strongly agreed with the statement, “I am confident that the Bookmark Procedure used produced valid cut scores.”
• 100% of participants agreed or strongly agreed with the statement, “I feel this procedure was fair.”
25
Participants’ Comments
• “It is satisfying to know the work done better reflects the rigor of “new” standards.”
• “It’s scary raising the scores, but it must be done to align nationally and internationally. I work in a small rural school with lots of special needs students—raising scores really hurts—but hopefully education policymakers will help us fund more help for smaller class sizes and special education case loads.”
26
Evaluation SummaryMet? Standard
✔ 1.Panels should be large enough and representative of the appropriate constituencies.
✔ 2.Selection and qualification of participants should be documented.✔ 3.Two panels or subpanels should be used to check the generalizability of the standards. ✔ 4.Background and demographic information about participants should be collected and documented.
✔5.To ensure internal validity, the methods must be consistent so that ratings indicate increased internal consistency across rounds and panelists.
✔ 6.To ensure procedural validity, the procedures must be reasonable, carried out as intended and understood by panelists.✔ 7.The methodology should be appropriate for the assessment, described in detail and field tested when appropriate.✔ 8.Any non-standard methodology must be clearly documented.✔ 9.The precise nature of participants’ judgments should be documented. ✔ 10.The rationale and procedures for establishing cut scores must be documented.
✔11.The methods should be designed so that participants can reasonably contribute their knowledge and experience to produce reasonable, defensible standards.
✔12.Participants should be suitably trained on the methodology; training should include a thorough description of the method and practice exercises, practice administration of the assessment, and practice judging task difficulty with feedback on accuracy.
✔ 13.Descriptions of performance categories must be clear to the extent that participants are able to use them effectively.✔ 14.The process should be conducted efficiently.✔ 15.Item booklets, rating forms and other provided documents should be easy to use.
✔16.Facilitators should be qualified and capable of leading appropriate discussion among the participants without biasing the process.
✔ 17.Feedback to participants must be clear, understandable, and useful.
✔18.Participants should be instructed on the appropriate use of provided data (including performance data, impact data, criterion reference data, etc).
✔ 19.When possible, performance levels should be established using empirical criterion reference data.✔ 20.Process evaluations should be conducted and documented.
✔21.The entire process must be documented, including participant selection and qualifications, training, feedback to panelists regarding their recommendations, replicability, validity and variability over participant recommendations.
27
Implications The recommended cut scores raise Oregon’s
standards to 4th (grade 4) and 5th (grade 8) highest in the country on the NAEP scale
• Raised expectations for teachers and students to some of the highest in the country
• On par with highly competitive international standards
• Improved prediction of success in college math courses
• Improved prediction of the probability of meeting high school standards, allowing more time for intervention
28
Implications, cont’d
• Innovative in the provision of external reference data for consideration when setting cut scores
• Feedback from participating stakeholders was positive:
• “It was a real pleasure working with you and the entire assessment team this week in Salem. I know that the principal who initially nominated me will be thrilled to learn that I participated. Thank you for that opportunity. ”
• “Thanks for including me in this process. It was enjoyable, thought provoking, and extremely helpful when viewed within the context of the work being done around math and CTE programs at the community college level. ”
