Where Did Your Cut Score Come From?: A Primer in Standard Setting
-
Upload
susan-metzger-gracia -
Category
Documents
-
view
215 -
download
0
Transcript of Where Did Your Cut Score Come From?: A Primer in Standard Setting
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 1/32
Click to edit Master subtitle style
March 27, 2012
Where Did Your Cut ScoreCome From?: A Primer in
Standard SettingDr. Susan GraciaDirector of AssessmentSimmons CollegeBoston, MA
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 2/32
March 27, 2012
Cut Score• A selected point on the score scale of
a test.•
The cut score separates students intovarious categories, such as a passingand failing.
•
The process of setting a cut score iscalled standard setting , which is amulti-stage, judgmental process.
22
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 3/32
March 27, 2012
Nomenclature• Achievement levels• Classification scores•
Criterion levels• Cut points• Cut scores• Cutscores• Cutting scores
• Cutoff scores• Mastery levels•
Passing scores• Performance levels• Performance standards• Standards (ETS, 2005)
33
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 4/32
March 27, 2012
Determine the Need for CutScores
• Why is there a need to set cutscores?
•
What benefits are expected from theuse of cut scores?• What decisions will be made on the
basis of the cut scores?• How are those decisions being made
now in the absence of cut scores?•
What reasons are there to believe 44
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 5/32
March 27, 2012
There is no “true” cut score• There’s no objective way to set cut
scores.•
Cut scores are constructed, notfound.• They are based on judgments about
people, student work, or test items.• Judges depend on subjective,
internalized norms about what
people can do. 55
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 6/32
March 27, 2012
Errors of Classification• Some people who deserve to pass
will fail.•
Some people who deserve to fail willpass.• Raising or lowering the cut score to
reduce one type of error will increasethe other type of error.• There’s no way to prove that a cut
score is correct. – “ ”
66
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 7/32
March 27, 2012
To Do for All Methods
1. Select judges.
2. Teach judges about cut scores.
3. Define “borderline” knowledge/skillsfor a particular task/assessment.
1. Borderline = Minimally competent
4. Train judges in the use of themethod.
5. Implement the cut score study.
6. Document the results. 77
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 8/32
March 27, 2012
Judges• Judges need to:
– Be qualified –
Know subject and population – Be representative and diverse – Be acceptable to stakeholders –
Be willing to follow procedures• Teach them about test purpose, cut
score purpose, consequences of passing and failing 88
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 9/32
March 27, 2012
Defining “Borderline”
1. Make sure judges understand whatthe assessment measures and howscores will be used.
2. Ask judges to describe in their ownwords a person whose knowledgeand skills would represent theborderline between acceptable andunacceptable levels of knowledge/skills the assessment
measures. 99
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 10/32
March 27, 2012
Examples of Borderlines• Worst reader who should get a
diploma / best reader who shouldnot
• Worst surgeon who should belicensed / best surgeon whoshould not
• Worst essay that deserves a score of 6 / best essay that deserves a 5
•
Highest bacteria count in safe water /lowest bacteria count in olluted1010
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 11/32
March 27, 2012
Judge Training• Overview of the standard setting
method: What is it? Why do it? Howis it done?
• Practice using the standard settingmethod.
•
Observe judges.• Correct errors.• Answer all questions.• Practice more until all judges are 1111
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 12/32
March 27, 2012
Selecting a Method• There’s no one best method.• Different methods will result in
different cut scores.• Some methods are better for certain
types of assessments and
assessment situations.
1212
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 13/32
March 27, 2012
3 Standard Setting MethodsMethods Based on
judgments about:Best
Modified Angoff Method
Test items Test that is only orprimarily multiplechoice
ContrastingGroups Method
People Small assessmentsituations wherefaculty havestrong knowledge
of students’abilities
Body of WorkMethod
Student work Performance tasks
1313
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 14/32
March 27, 2012
Modified Angoff Method• Judges examine each question on an
assessment and estimate theprobability that a borderline studentwould answer the question correctly.
– Or: Imagine a group of 100 borderlinestudents and estimate how many of
them would answer the questioncorrectly.• Probabilities will range from .00 to
1.00. 1414
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 15/32
March 27, 2012
Modified Angoff Method• Discuss probabilities. Aim for range
of 10-15 % points per question. Judges can change judgments basedon discussion.
• Sum each judge’s probabilities for allquestions to get a recommended cutscore for each judge.
• Average the judges’ recommendedcut scores to arrive at a average cutscore for minimum competency. 1515
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 16/32
March 27, 2012
Modified Angoff Example
1616
• Judge 1
Source: Livingston &
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 17/32
March 27, 2012
Modified Angoff Example Judge Recommended Cut Score
1 5.80
2 6.00
3 6.00
4 5.405 5.00
6 5.30
7 5.50
8 4.80
9 6.10
10 5.50
Average Cut Score forMinimum Competency
5.54
1717
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 18/32
March 27, 2012
Modified Angoff • Pros• Most researched
method• Stands up to
court challenges• Does not require
student data soit can be carriedout prior to testadministration
• Cons• A lot of dataentry
• Difficultcognitive task
• Does not workwell with heavilyopen-ended/performance-based tests
1818
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 19/32
March 27, 2012
Contrasting Groups Method• Faculty consider all they know about
a population of students.• Faculty predict individual students’
level of performance on anassessment (e.g., expectedpassing/failing, proficient/non-proficient) –without reference toscores.
• Obtain assessment scores.• Predictions are com ared to actual 1919
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 20/32
March 27, 2012
Contrasting Groups Example
2020
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 21/32
March 27, 2012
Another Example• Graph the scores of expected passing
and expected failing students in 2separate distributions.
• The point at which the 2 distributionsintersect is the cut score.
2121
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 22/32
March 27, 2012
Contrasting Groups• Pros• Uses real data,
not conjecture• Uses external
validation info inaddition to testscores
• Easy to explain
• Cons• Inconvenient toget judgments of people
• Relies on human judgmentwithout
examining actualtest performance
• Need scores
before cut is set2222
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 23/32
March 27, 2012
Body of Work Method• Pre-work:
– Performance level categories, as well asperformance level descriptors for eachperformance level, must be establishedand agreed upon.
– Scoring of a large sample of student
work must occur before standard-settingcan begin. – Select 40 to 50 intact samples of
student work to represent the range of student performance on an assessment.2323
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 24/32
March 27, 2012
Body of Work Method• Range-Finding phase:
– Identical sets of student work areprovided to each judge.
– Judges are asked to independentlycategorize the student work samplesbased on the performance level
descriptors, without any discussion. – This process reveals which work
samples (e.g., Graduation Portfolios)generate the most agreement and whichgenerate the most disagreement 2424
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 25/32
March 27, 2012
Examination of StudentWork Method
• Pinpointing phase: – Judges examine sets of work about
which they disagreed in the range-finding phase, along with additionalwork samples representing those samescore intervals.
–
Judges assign performance levels tothese work samples. – The minimum score for each
performance level is precisely"pinpointed" by determining the score 2525
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 26/32
March 27, 2012
Body of Work• Pros• Uses real data,
not conjecture• More intuitive;
panelists are notasked to imaginea hypothetical
minimallycompetentexaminee or to
estimate the
• Cons• Need scoresbefore cut is set
• Requires a lot of priorpreparation,volumes of
materials• Grueling work
2626
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 27/32
March 27, 2012
Setting the “Operational”Cut Score
• Only a legally authorizedentity (e.g., policy makers)
can authorize the use a cutscore.
•
Once authorized, a “study”cut score becomes an“operational” cut score.
2727
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 28/32
March 27, 2012
Complying with theStandards
• The Standards of the AmericanEducational Research Association,the American PsychologicalAssociation, and the National Councilon Measurement in Education specifythat the following be included in the
standard setting process: – Document how judges were selected,
their qualifications and training, theprocedures used, whether or not judges’ratings were independent, the level of 2828
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 29/32
March 27, 2012
Observe Cut Score Effects• Seek opinions on cut scores.• Find out what happened to people
who failed. – Is there evidence that any of them were
actually qualified?•
Is there evidence that any peoplewho passed are really unqualified?• What were the consequences of
misclassification errors? 2929
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 30/32
March 27, 2012
Comments? Questions?
3030
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 31/32
March 27, 2012
Discussion Questions• How could you use standard setting
in your setting?• Which standard setting approach
might you utilize? Why?• What challenges do you anticipate in
implementing this approach?• How might you address these
challenges?
3131
7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting
http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 32/32
March 27 2012
References
3232
American Educational Research Association , American Psychologica l Association ,National Council on Measurement in Education. ( ).1111 Standards for Educational andPsychological Testing. Washington, DC: AmericanPsychological Association.
Cizek, G.J. ( ). An NCME instructional module on setting pass scores.1111 Educational Measurement: Issues and Practice , 11 ( ),1 11 - .11
Cizek, G.J., Bunch , M.B., Koons, H. (Winter ). Setting performance standards:1111 Contemporary methods. Educational Measurement: Issues and Practice , 11 - .11
Horn, C., Ramos, M., Blumer, I. & Madaus, G. (1111 ). Cut scores: Results may vary.National Board on Educational Testing and Public Policy Monographs,1 ( ).1 Chestnut Hill, MA: Boston College.
Livingston, S.A. & Zieky, M.J.. ( ).1111 Excerpts from Passing Scores: A Manual for Setting Standards of Performance on Education and Occupational Tests .Princeton,NJ: Educational Testing Service.
Measured measures: Technical considerations for developing a local assessment system . ( ). Augusta , ME: Maine Department of Education.1111 Pitoniak, M. ( ).1111 Considerations in Setting Performance Standards ( Cutscores).
Training session at 1111 National Council for Measurement in Educationconference, Montreal, Canada.