Post on 25-Dec-2015
“Some” Implications of Large Scale Assessment
for MSPs
Marge PetitMarge Petit
Center for AssessmentCenter for Assessment
May 2004May 2004
Accountability that only Accountability that only focuses on identifying focuses on identifying schools is not going to schools is not going to improve schools. improve schools. Brian GongBrian Gong
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 2
Some Definitions
Large-scale assessment;Large-scale assessment; Standards; Standards; Standards-based large-scale assessment;Standards-based large-scale assessment; Grade level expectations; andGrade level expectations; and Grade level assessments.Grade level assessments.
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 3
NCLB and Challenges (Jim Pelligrino, NRC Conference 2004)
§ Designing and implementing assessment systems to respond effectively to the multiple provisions of No Child Left Behind;
§ Defining the appropriate targets for assessment relative to content standards;
§ Determining how targets might be apportioned across different elements of a comprehensive assessment system; and
§ Developing and supporting assessment systems that provide information to support the enhancement of learning and instruction as well as accountability.
The multiple sets of goals and needs is a major conceptual and operational design challenge.
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 4
Primary Goal of MSPs
To improve student learning in mathematics and science!
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 5
Givens All schools associated with MSPs are responsible for improving All schools associated with MSPs are responsible for improving
student learning in mathematics and science as they relate to state student learning in mathematics and science as they relate to state standards and grade level expectations;standards and grade level expectations;
All schools associated with MSPs are responsible for improving All schools associated with MSPs are responsible for improving student performance on state assessments;student performance on state assessments;
All schools are responsible for making Adequate Yearly Progress All schools are responsible for making Adequate Yearly Progress in mathematics as defined by each state consistent with in mathematics as defined by each state consistent with requirements of NCLB;requirements of NCLB;
The state level assessment is currently the basis for determining The state level assessment is currently the basis for determining AYP;AYP;
One measure of the effectiveness of your MSP will probably be One measure of the effectiveness of your MSP will probably be an increase in the percentage of student at proficient or above on an increase in the percentage of student at proficient or above on state grade level assessments.state grade level assessments.
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 6
Implications for MSPs The quality of grade level assessments will impact your work and The quality of grade level assessments will impact your work and
mathematics and science instruction and learningmathematics and science instruction and learning . . Learn about the Learn about the issues facing your state, and get involved in the ongoing issues facing your state, and get involved in the ongoing development of the grade level assessments in your state.development of the grade level assessments in your state.
The quality of grade level expectations will impact your work and The quality of grade level expectations will impact your work and the quality of mathematics and science instruction and learning. the quality of mathematics and science instruction and learning. All All that is valued in mathematics and science may or may not be articulated that is valued in mathematics and science may or may not be articulated in state standards and grade level expectations. in state standards and grade level expectations. Find the gaps and fill.Find the gaps and fill.
Large scale assessment CANNOT provide ALL the information that Large scale assessment CANNOT provide ALL the information that is needed to improve mathematics/science instruction and learning.is needed to improve mathematics/science instruction and learning. Build a “coherent and balanced” assessment system to support Build a “coherent and balanced” assessment system to support MSP schoolsMSP schools..
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 7
Implications for MSPs The quality of grade level assessments will impact your work and The quality of grade level assessments will impact your work and
mathematics and science instruction and learningmathematics and science instruction and learning . . Learn about the Learn about the issues facing your state, and get involved in the development of the issues facing your state, and get involved in the development of the grade level assessments in your state.grade level assessments in your state.
The quality of grade level expectations will impact your work and The quality of grade level expectations will impact your work and the quality of mathematics and science instruction and learning. the quality of mathematics and science instruction and learning. All All that is valued in mathematics and science may or may not be articulated that is valued in mathematics and science may or may not be articulated in state standards and grade level expectations. in state standards and grade level expectations. Find the gaps and fillFind the gaps and fill..
Large scale assessment CANNOT provide ALL the information that Large scale assessment CANNOT provide ALL the information that is needed to improve mathematics/science instruction and learning.is needed to improve mathematics/science instruction and learning. Build a “coherent and balanced” assessment system to support Build a “coherent and balanced” assessment system to support MSP schoolsMSP schools..
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 8
“Some” Real time - Practical and Political issues interacting with technical and educational issues
Alignment;Alignment; Testing time – time of year;Testing time – time of year; Turn-around time and the impact on item types;Turn-around time and the impact on item types;
Item types and Depth of Knowledge;Item types and Depth of Knowledge; Integrity of the content;Integrity of the content; Level of information derived;Level of information derived;
Access for the greatest number of students;Access for the greatest number of students; Science LiteScience Lite Standard setting; and Standard setting; and Implementation cost.Implementation cost.
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 9
Dimensions of Alignment (e.g., Webb, 1997;
Achieve, Inc., Porter, 2002; WestEd)
ContentContent Cognitive demand (e.g., Depth of Cognitive demand (e.g., Depth of
Knowledge (Webb); Level of Challenge Knowledge (Webb); Level of Challenge (Achieve)(Achieve)
Balance;Balance; RangeRange
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 10
Porter Alignment on 2 dimensions
Grade 8Grade 8
MemorizeMemorize Perform Perform
ProceduresProcedures
Communicate Communicate UnderstandingUnderstanding
Solve non-Solve non-routing routing
problemsproblems
Conjecture, Conjecture, Analyze, ProveAnalyze, Prove
SimilaritySimilarity
Pythagorean Pythagorean TheoremTheorem
Porter alignment dimensions
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 11
12 ft
8 ft
A
B C
What is the length of ?AC
Range of items
Wyoming BOE
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 13
How much alignment is enough?
A: The state standards and state assessment should be perfectly matched/aligned. (True or False – Why or why not?)
State Standards/grade level expectations
State Standards/grade level expectations
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 14
B: The state standards should be a subset of the objectives assessed on the state assessment.
State Assessment
State Standards/grade level expectations
How much alignment is enough?
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 15
The objectives assessed on the state assessment should be a subset of the state standards/grade level expectations.
How much alignment is enough?
State Standards/grade level expectations
State Assessment
C
State Standards/grade level expectations
State Assessment
D
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 16
State Standards/Grade Level Expectations
Mentally add and subtract whole numbers Mentally add and subtract whole numbers to ….to ….
Design and conduct an experiment to Design and conduct an experiment to answer a student or teacher generated answer a student or teacher generated experiment…experiment…
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 17
Alignment
Grade 8Grade 8
MemorizeMemorize Perform Perform
ProceduresProcedures
Communicate Communicate UnderstandingUnderstanding
Solve non-Solve non-routing routing
problemsproblems
Conjecture, Conjecture, Analyze, ProveAnalyze, Prove
Lines, slopes Lines, slopes and and interceptsintercepts
xx xx xx
Local Curriculum: Students will demonstrate conceptual understanding of linear relationships as a constant rate of change by determining the slope of a line, explaining the meaning of slope and intercept in context from a table,graph or situation, and by solving routine and non-routine problems involving the relationship between the rate of change, slope, and intercept.
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 18
Two Types of Grade Level Expectations
Test Specification for Test Specification for the large scale the large scale assessmentassessment
Specification for local Specification for local curriculum and curriculum and assessment.assessment.
Test Specification
for large-scale
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 19
Some Characteristics of Two Types
Test SpecificationTest Specification Must be assessable in an Must be assessable in an
on-demand large-scale on-demand large-scale setting;setting;
Should be a prioritized set.Should be a prioritized set.
Local Curriculum and Local Curriculum and AssessmentAssessment
Can include concepts and Can include concepts and skills not easily assessable skills not easily assessable in an on-demand setting;in an on-demand setting;
Can include foundational Can include foundational skills as they develop skills as they develop across grades.across grades.
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 20
Examples
Test SpecificationTest Specification M–DSP–3–1M–DSP–3–1 Interprets a Interprets a
given representationgiven representation ((line line plots, bar graphs, tally charts, plots, bar graphs, tally charts, or tables) to answer questions or tables) to answer questions related to the data, to analyze related to the data, to analyze the data to formulate the data to formulate conclusions, or to make conclusions, or to make predictionspredictions.. (New England (New England Common Assessment Program)Common Assessment Program)
Local Curriculum and Local Curriculum and AssessmentAssessment
GLE #M26: In response to a GLE #M26: In response to a teacher or student generated teacher or student generated question or hypothesis, question or hypothesis, collects appropriate data, collects appropriate data, organizes the data, organizes the data, displays/represents the data and displays/represents the data and makes observations about the makes observations about the data to draw conclusions about data to draw conclusions about the questions or hypothesis the questions or hypothesis being tested. (Local Vermont being tested. (Local Vermont GLE)GLE)
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 21
Why a balance of item types?
•To adequately assess GLEs within a given time period;•To appropriately assess the content and cognitive demand in GLEs; and •To appropriately reflect instructional and assessment practices consistent with GLEs.
“Large scale assessments should be substantially consistent with high quality classroom assessments though procedurally separate.” Shepard 2000
Link to item type instruction
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 22
Item Types“(1) you get what you assess... [and] (2) you do not get what you do not assess... (Resnick & Resnick,
1992, p. 59)
Write a rule using symbols for the nth term in the following arithmetic sequence.
Modified - Texas 2002 8th grade
Generalize a variety of patterns, and represent the
patterns symbolically.
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 23
Generalize a variety of patterns, and represent the
patterns symbolically.
Item Types“(1) you get what you assess... [and] (2) you do not get what you do not assess... (Resnick & Resnick, 1992, p. 59
Texas 2002 8th grade
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 24
Science “Lite” Ed Week Jan. 2004 Science “Lite” Science “Lite”
Items that don’t require scienceItems that don’t require science
Item with technical design flawsItem with technical design flaws
39 state study - Carnine, Miller Metzenberg
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 25
Science “Lite” Ed Week Jan. 2004
Doesn’t require Doesn’t require sciencescience
A lever is a bar that turns around a fixed point called a fulcrum. A pair of scissors is made of two levers that move in opposition [an illustration of a pair of scissors are provided with A – D]. Which of the following points is the fulcrum of the two levers?
A) Point AB) Point BC) Point C
D) Point D
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 26
Science “Lite” Ed Week Jan. 2004
Science “Lite” Science “Lite”
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 27
Access for the greatest number of students … … provide the greatest number of students the provide the greatest number of students the
opportunity to demonstrate their knowledge and opportunity to demonstrate their knowledge and skills in relationship to the skills in relationship to the mathematical or mathematical or science constructscience construct being assessed. being assessed.
Universal Design
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 28
Conserving the Mathematical Construct (Petit and Lager 2003)
Explicitly aligning items with mathematical Explicitly aligning items with mathematical construct being assessed– content and process construct being assessed– content and process demands in GLE;demands in GLE;
Make intentional decisions of when and how to Make intentional decisions of when and how to use context to assess the mathematics constructuse context to assess the mathematics construct;;
Streamlining language (Lager, 2003) to provide Streamlining language (Lager, 2003) to provide access without compromising mathematical access without compromising mathematical construct being assessed;construct being assessed;
Appropriately using graphics, pictures, graphs, Appropriately using graphics, pictures, graphs, tables, diagrams, and models.tables, diagrams, and models.
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 29
Implications for MSPs The quality of grade level assessments will impact your work and The quality of grade level assessments will impact your work and
mathematics and science instruction and learningmathematics and science instruction and learning . . Learn about the Learn about the issues facing your state, and get involved in the development of the issues facing your state, and get involved in the development of the grade level assessments in your state.grade level assessments in your state.
The quality of grade level expectations will impact your work and The quality of grade level expectations will impact your work and the quality of mathematics and science instruction and learning. the quality of mathematics and science instruction and learning. All All that is valued in mathematics and science may or may not be articulated that is valued in mathematics and science may or may not be articulated in state standards and grade level expectations. in state standards and grade level expectations. Find the gaps and fill.Find the gaps and fill.
Large scale assessment CANNOT provide ALL the information that Large scale assessment CANNOT provide ALL the information that is needed to improve mathematics/science instruction and learning.is needed to improve mathematics/science instruction and learning. Build a “coherent and balanced” assessment system to support Build a “coherent and balanced” assessment system to support MSP schoolsMSP schools. .
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 30
At the foundation of any assessment or accountability system based on student performance is the framework of concepts and skills upon which they are to be built.
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 31
The ultimate validity test…
“If a school is identified under NCLB for low performance in mathematics, the response by the school should be to strengthen the mathematics instruction based upon a solid
set of grade level expectations.” Tim Kurtz, NH Assessment Director
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 32
Primary Goal of MSPs
To improve student learning in mathematics and science!
MSP Responsibility: Go beyond the bullets on the GLEs to support curriculum, instruction, and assessment with deep and lasting understanding of concepts and skills.
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 33
New Jersey Grade 2---By the end of Grade 2, students will:A. Number Sense1. Use real-life experiences, physical materials, and technology to construct meanings for numbers (unless otherwise noted, all indicators for grade 4 pertain to these sets of numbers as well).
•Whole numbers through millions• Commonly used fractions (denominators of 2, 3, 4,5, 6, 8, 10, 12, 16)
2. Demonstrate a sense of the relative magnitude of numbers. 3. Understand the various uses of numbers; 4. Use concrete and pictorial models to relate whole numbers, commonly used fractions, and decimals to each other, and to represent equivalent to forms of the same number;5. Compare and order whole numbers.6. Compare and order numbers.7. Explore settings that give rise to negative numbers.
What does it mean to develop deep and lasting
understanding of…. ?
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 34
By the end of Grade 4, students will:A. Number Sense1. Use real-life experiences, physical materials, and technology to construct meanings for numbers (unless otherwise noted, all indicators for grade 4 pertain to these sets of numbers as well).
•Whole numbers through millions• Commonly used fractions (denominators of 2, 3, 4,5, 6, 8, 10, 12, 16)
2. Demonstrate a sense of the relative magnitude of numbers. 3. Understand the various uses of numbers; 4. Use concrete and pictorial models to relate whole numbers, commonly used fractions, and decimals to each other, and to represent equivalent to forms of the same number;5. Compare and order whole numbers.6. Compare and order numbers.7. Explore settings that give rise to negative numbers.
What does it mean to develop deep and lasting
understanding of…. ?
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 35
An Example – Does this sequence promote deep and lasting understanding
Shade 1/3 of the figure.
What is 1/3 of 27?
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 36
Students progress through four levels of Students progress through four levels of partitioning: sharing; algorithmic halving; partitioning: sharing; algorithmic halving; evenness; oddness; and composition. (Pothier and evenness; oddness; and composition. (Pothier and Sawada, 1990, cited in Bezuk and Bieck, 1993, Sawada, 1990, cited in Bezuk and Bieck, 1993, pp.124 – 125)pp.124 – 125)
““Fair share” means the same number of pieces or Fair share” means the same number of pieces or parts regardless of their size. Partitions are drawn parts regardless of their size. Partitions are drawn in a shape for the correct number of parts, in a shape for the correct number of parts, disregarding the size of the parts (e.g., Cutting a disregarding the size of the parts (e.g., Cutting a circle with four vertical lines). (Payne, 1975) circle with four vertical lines). (Payne, 1975)
•Proper fractions (denominators of 2, 3, 4, 8, 10)
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 37
Shared with permission of the Vermont Mathematics Partnership
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 38
Questions about GLEs and local curriculum
Do the GLEs promote and support quality Do the GLEs promote and support quality instruction and assessment in mathematics and instruction and assessment in mathematics and science?science?
Have aspects of science and mathematics not Have aspects of science and mathematics not easily assessed in large scale, but important, been easily assessed in large scale, but important, been included for local curriculum and assessment?included for local curriculum and assessment?
Is local curriculum and assessment aligned with Is local curriculum and assessment aligned with state GLEs?state GLEs?
Are the GLEs and local curriculum prioritized?Are the GLEs and local curriculum prioritized? Other?Other?
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 39
Implications for MSPs The quality of grade level assessments will impact your work and The quality of grade level assessments will impact your work and
mathematics and science instruction and learningmathematics and science instruction and learning . . Learn about the Learn about the issues facing your state, and get involved in the development of the issues facing your state, and get involved in the development of the grade level assessments in your state.grade level assessments in your state.
The quality of grade level expectations will impact your work and The quality of grade level expectations will impact your work and the quality of mathematics and science. the quality of mathematics and science. All that is valued in All that is valued in mathematics and science may or may not be articulated in state standards mathematics and science may or may not be articulated in state standards and grade level expectations. and grade level expectations. Find the gaps and fill the, but doFind the gaps and fill the, but do not not support “mile wide and inch deep” curriculum or assessment. Help support “mile wide and inch deep” curriculum or assessment. Help Prioritize!Prioritize!
Large scale assessment CANNOT provide ALL the information that Large scale assessment CANNOT provide ALL the information that is needed to improve mathematics/science instruction and learning.is needed to improve mathematics/science instruction and learning. Build a “coherent and balanced” assessment system to support Build a “coherent and balanced” assessment system to support MSP schoolsMSP schools..
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 40
Large Scale Assessments are Not valuable for… Lorrie Shepard, Assessment in Support of Learning
Providing detailed understanding of Providing detailed understanding of individual student learning on an ongoing individual student learning on an ongoing basis!basis!
Capturing synthesis of knowledge or some Capturing synthesis of knowledge or some concepts that are not easily assessed in concepts that are not easily assessed in large-scale assessment.large-scale assessment.
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 41
Local Assessment: Webb, DoK Level 4 – Curriculum Embedded Assessment
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 42
UsesUses Level of Level of InfluenceInfluence
Large Scale State Level Large Scale State Level AssessmentAssessment
Local – Large Scale Local – Large Scale AssessmentAssessment
Classroom summative Classroom summative assessmentassessmentTests, quizzes, projects, etc.Tests, quizzes, projects, etc.
Formative AssessmentFormative Assessment
OtherOther
How balanced is the assessment system in MSP schools?
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 43
Characteristics of a Coherent and Balanced
Assessment System
• Comprehensive; (Adapted from Knowing What Student Know(NRC, 2001)
• Coherent; (Adapted from Knowing What Student Know(NRC, 2001)
• Continuous; (Adapted from Knowing What Student Know(NRC, 2001)
• Integrated; (Distilled from other reports listed on last page)
• Includes quality assessments. (Distilled from other reports
listed on last page)
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 44
What do I know? What do I need to find out? What actions should I take? Assessment design issues facing my state;Assessment design issues facing my state; The design of grade level assessments;The design of grade level assessments; The nature of school and student level information The nature of school and student level information
derived from large scale assessment;derived from large scale assessment; The status of the grade level expectation The status of the grade level expectation
development and implementation;development and implementation; The degree to which educators have the range of The degree to which educators have the range of
information needed to make program adjustments information needed to make program adjustments and to make instructional decisions – “on time”;and to make instructional decisions – “on time”;
May 2004 Petit - v. 9 Center for Assessment for the National Reserach Council 45
References
Bezuk, N. S., and Bieck, M. (1993). Current Research on Rational Numbers and Common Fractions: Summary and Implications for Teachers. In D.T. Owens (Ed.), Research Ideas for the Classroom: Middle Grades Mathematics (pp. 118-136). New York: Macmillan.
Lager, C., Petit, M., (2003) Conserving the Mathematical Construct, TSNE Test Specifications v. 12,,
National Research Council (2001a), Adding it Up: How Children Learn Mathematics. Mathematics Learning Study Committee, Jeremy Kilpatrick, Jane Swan, and Bradford Findell (Eds.)., Washington D.C., National Academy Press.
National Research Council (2003), Assessment in Support of Student Learning: Bridging the Gap Between Large-scale and Classroom Assessment. Mathematical Science Education Board, Board on Testing and Measurement, and Committee on Science Education Washington D.C., National Academy Press.
National Research Council (2001b), How People Learn: Brain, Mind, Experience, and School. Committee in the Development of the Science of Learning. John Bransford, Ann Brown, and Rodney Cocking (Eds.) Washington D.C., National Academy Press.
National Research Council (2000), Knowing What Students Know: The Science and Design of Educational Assessment. Committee on the Foundations of Assessment. James Pelligrino, Naomi Chudowsky, and Robert Glaser (Eds.) Washington D.C., National Academy Press.
National Research Council (2000), Learning about Assessment, Learning Through Assessment,. Mathematical Sciences Education Board. Mark Driscol and Deborah Bryant (Eds.) Washington D.C., National Academy Press.
National Research Council (2000), Classroom assessment and the national science education standards, . J. Myron Adkins, Paul Black, and Janet Coffey (Eds.) Washington D.C., National Academy Press.
Payne, J. N.. (1975) Review of research on fractions. In R. Lesh and Bradbard (Eds.) Number and measurement: Papers from a research workshop. (Page 145 – 187), Columbus, Ohio.
Petit, M., (2003) Aligning to What?, RILS Annual Meeting. www.nciea.orgPopham, J. [In press} (2003) Crafting Curricula Aims for Instructionally Supportive AssessmentWebb, N., (2002)Resnick, L. B., & Resnick, D. P. (1992). Assessing the thinking curriculum: New tools for educational reform. In B. R. Gifford & M. C. O'Conner (Eds.), Changing assessments: Alternative views of aptitude, achievement and instruction. Boston: Kluwer Academic