Standard Setting Procedures
-
Upload
rizwan-zafar-ansari -
Category
Documents
-
view
53 -
download
3
description
Transcript of Standard Setting Procedures
RESEARCH METHODOLOGY
Procedures for Establishing Defensible Absolute Passing Scores on
Performance Examinations in Health Professions Education
Steven M. DowningAra Tekian
Rachel YudkowskyDepartment of Medical Education
University of Illinois at ChicagoChicago, Illinois, USA
Learning Objectives
• By the end of IP we should able to • Describe standard setting methods• Differentiate b/w their types
– Norm based – Criterion based – Relative – Absolute
• Know Passing Score• Describe selection of Judges (examiners)• Identify Borderline Examinee• Understand each method
What do experts say?
"We have come to realize that there is no objectively correct way to set standards. But we have also come to realize that there is nothing wrong with using judgments appropriately." (Zieky, 1995, p.5)
"Determination of a minimum acceptable performance always involves some rather arbitrary and not wholly satisfactory decisions." (Ebel, 1972, p.492)
Why we need standard setting methods?
– To determine the standards of performance– To separates the Non-competent from the
Competent– To provide an educational tool to decide CUT OFF
POINT on the score scale
• (Reference AMEE Guide No. 18: Standard setting in student assessment)
essentials
• 1. choice of content expert 5-6 or 11- 12
• 2. identification of borderline examine
• 3. cut score
Choice of content expert
Judges should be to• Judge examinee performance Unbiased• Follow the Instructions • Understand their taskJudges should be • subject experts• Belong from variable culture, ethics, religion and
both genders ( male and female judges)• 5 -12 judges panel is better.
Borderline examinee
• One who has 50 50 probability of passing or failing the test.
• Sometimes passes the exam and sometimes fails
Or • Judges will decide the characteristics of
borderline examinee
Cut Score
• There is no gold standard for pass scores. • Passing score is what ever the judges decide.• Different panel of judges may decide different
passing score for the same exam.• It depends upon how much is enough to pass,
the subject experts will decide by devising a check list / predetermined key or left it open on judges judgment.
Problem with judges
• Judges expect even more from borderline examinee
• They set unrealistically high standards which fails a reasonably high proportion of examinee e.g– in viva(Examiners expectations are so high ) It happens when judges decide cut off without
knowing the actual performance data.
How can we overcome this problem• Celebration of judges by
– Providing student record showing their overall performance.
– Student record
Test Topics MCQs SAQs OSPE VIVA
Jan – Feb Thanatology Fail Fail below Fail
Mar – April Autopsy and exhumation Pass Fail Pass Fail
May – June Asphyxia Pass Pass Pass Borderline
Aug – Sept PI and traumatology Above Above Excellent Pass
Send up Above pass Very good Good
Passing Rate
no of passed students x 100 total no of students appeared
64/100x100 = 64%
0Not done
1poor
2 3 Below expectation
4 borderline
5 6 Meet expectation
7 Above expectation
4 6 4 17 24 35 7 3
PASSED = 64
Standards setting Methods
Relative
Item based
Performance based
Relative
Item based
Modified Angoff Ebel
performance
Original Angoff
Angoff Method• Judgments of the judges are combined to
determine passing score• Rater• Item 1 2 3 4 5 6 7 M• 1 .80 .87 .85 .90 .80 .95 .85 0.86• 2 .70 .75 .80 .85 .75 .85 .75 0.78• 3 .50 .63 .55 .60 .65 .60 .60 0.59• 4 .70 .68 .70 .70 .65 .70 .70 0.69• 5 .75 .70 .80 .85 .70 .85 .80 0.78• 6 .60 .65 .80 .75 .65 .85 .80 0.73• 7 .50 .58 .55 .60 .70 .90 .60 0.63• 8 .70 .78 .75 .75 .65 .80 .70 0.73• 9 .45 .50 .50 .45 .43 .55 .45 0.48• 10 .60 .69 .65 .65 .65 .70 .70 0.66• Sum 6.93• Pass Score is 69.30%• Raw Passing Score = Sum of item means = 6.93. Percent Passing• Score = 100% × (sum of item means/number of items) = 100% ו (6.93/10) = 69.30%.
Angoff Passing Score
Angoff’s method - 2
– Read the first item– Estimate the proportion of the borderline group that
would respond correctly– Record ratings, discuss, and change – Repeat this for each item– Calculate the passing score by
• adding rating score of each item separately (modified angoffs) e.g FCPS examinee has to satisfy all judges
• Adding performance of all stations (original angoff) OSCE
Ebels Method
• Judges define the check list and rating scale• Categorize items like essential, important,
acceptable• Rate item like easy medium hard• judges define the borderline performance to
pass (0 – 100 %)
Ebel’s method
Easy Medium Hard
Essential
Important
Acceptable
Ebel’s Method
– Judges make judgments about the percentages of items in each category that borderline test-takers would have answered correctly
– Calculate passing score
Ebel’s method %(borderline perform correctly)
Easy Medium Hard
Essential 95% 60% 40%
Important 90% 56% 34%
Acceptable
80% 60% 50%
Items Relevance
Easy Medium Hard Weighted Mean
Item # % correct
Item # % correct
Item # % correct
Essential 4 , 5 93 1 81 3 63 2(.93)+.81+.63= 3.30
Important 2 89 10 76 9 59 .89+.76+.59= 2.24
acceptable N/A N/A 7 62 6 , 8 42 .62+2(.42)= 1.46
T. Mean 3.30 + 2.24 + 1.46 = 7
Passing rate = Total mean x 100 / no of items = 7 x 100 / 10 = 70%
% correct is the mean judgment of all the judges , borderline examinee did correct.
Absolute
Criterion based
Norm based
• Criterion referenced methods :
– Based on how much the examinees know– Candidates pass or fail depending on whether they meet
specified criteria – In Criterion-referenced tests (or CRTs) performance of each
examinee is compared to a pre-defined set of criteria or a standard. The goal with these tests is to determine whether or not the candidate has the demonstrated mastery of a certain skill or set of skills.
– E.g . A national board medical exam is an example of a CRT. Either the examinee has the skills to practice the profession, in which case he or she is licensed, or does not.
– e.g. examinees must correctly answer 70% of the questions
• REF : NORM-REFERENCED VS. CRITERION-REFERENCED TESTING• May 22nd, 2008 by Danielle, Director of Sales and Marketing, Language Testing
Criterion referenced standard
50 %
Test score distribution (average group)
Test score distribution (good group)
Test score distribution (poor group)
• criterion based– Based upon already set criteria – e.g– 33% passing score in FA,BA exams– 50% passing score in MBBS exams – 60% passing score at post graduation level– 80% passing score in skilled exams.
Criterion based
borderline contrast
Contrasting Groups
• Performance is judged by check list or rating scale.
• Students are divided into expert and non-expert groups based on rating scale
• Graphical presentation .• Passing score is set at the insertion of two
distributions false positive and false negative.
Compromise Methods
• Advantages– Easy to implement– Educators are comfortable with the decisions
• Disadvantages– The cut score may not be in the area defined by
the judges’ estimates– The method is not the first choice in a high stakes
testing situation
Borderline Group• Examinee centered.• Performance of the examine is judge overall.• Faculty directly observe the performance• E.g OSCE• Each judge observe multiple examinee on same
station.• Judges use global rating scale • 1= fail, 2= borderline, 3= pass• The mean checklist score of borderline
examinee becomes the passing score.
Types of Standards
• Norm referenced methods (NTR ) :– Based on a comparison among the performances of examinees
Or– Compare examinee performance to that of other examinees. – Standardized examinations such as the SAT are norm-referenced
tests. The goal is to rank the set of examinees so that decisions about their opportunity for success (e.g. college entrance) can be made.
– e.g Normal distribution bells curve. A set proportion of candidates fails regardless of how well they perform e.g. the top 84% pass
• Norm based: • Cut off score is not pre defined
– Identify a group of passing and failing examinee bycomparing their performance– In OSCE there are 10 stations total Score 100, No
of examinee is 05
Mean = 320/5x = 64• 64 is cut off b/w pass and fail(REF: Medical Knowledge Using Progress Tests A.M.M.
Muijtjens, R.J.I. Hoogenboom, G.M. Verwijnen, C.P.M. van der Vleuten)
examinee 1 2 3 4 5
score 100 50 70 90 80 30
Norm-referenced standard
Test score distribution
30 %
50 % 80 %
Hofstee Method(relative – absolute compromise method)
• Judges are ask to define minimum and maximum passing score and failure rate .e.g
• 81 -100 % outstanding• 71 – 80 % above expectation• 61 – 70 % (max pass score) meet expectation• 56 – 60 % top borderline• 51 – 55 % (min pass score) bottom borderline• 40 – 50 % below expectation• 20 – 39 % perform incorrect• 0 - 19 % don’t know
Hofstee Method
• Graphical presentation• Judges predefined
– Fail rate e.g min 6, max 20 students to fail.– Acceptable
• Lowest pass score %• Highest pass score %
Min/max pass score
J – 1 J - 2 J - 3 J - 4 J - 5 J - 6 mean
Min 62 57 51 55 52 59 56
max 72 67 73 65 60 71 68
Hofstee Graph
Actual score
Min Max pass %
56 % 68%
61%
Max fail rate 20 %
Min fail rate 06 %
Cumulative %
Scores
Compensatory Vs Non compensatory
Compensatory• Poor performance on one
station can be compensated by good performance on other stations.
• Overall score will be the avg of performance on all the station.
• E.g SAQs, MMI, OSCE
Non compensatory• Student should reach the
minimum level of competence on each station.
• Student has to meet a predefined criteria on each station to pass.
• E.g OSATS, DOPS, Mini CEX
comparisonJudgment focused on
Judgment require performance data
Direct observation
Timing of judgememt
Angoff Test items / Performance
No No Before exam
Ebel Test items Yes No After exam
Hofstee Whole test Yes No After exam
Border line Examinee performance
No Yes During exam
Contrast Examinee performance
No Yes During exam
Summary :-
1. All standard-setting is judgmental
2. Standard-setting leads to errors of classification
3. Standard-setting is and will remain controversial
4. There is no purely absolute standard.
5. There is no one right method
6. Choosing judges is more important than
choosing methods
7.
Summery …….
• If the expert use rating scale or check list for assessment then you can choose borderline or contrasting method.
• If you don’t have expert rating the exam then you can choose Angoff, eble or Hofstee method
Critique
• This article describes only the standard settings for performance based exam ie OSCE, OSATS, DOPS
• Classification of standards is some what confusing.• Standards are overlapping no clear demarcation• These methods can be applied with some
modifications.• Dose not discuss percentile method.
References• AMEE guide No. 18• Berk, R.A. (1986). A consumer's guide to setting performance standards on
criterion-referenced tests. Review of Educational Research, 56, 137-172.
• Cizek, G. J. (2001). Setting Performance Standards: Concepts, Methods, and Perspectives. Mahwah, NJ: Lawrence Erlbaum Associates.
• Jaeger, R.M. (1989). Certification of student competence. In R.L. Linn (Ed.), Educational Measurement. New York: American Council on Education and Macmillan Publishing Company.
• Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425-461.
• Livingston, S.A. and Zeiky, M.J. (1982). Passing scores: A manual for setting standards of performance on educational and occupational tests. Princeton, NJ: Educational Testing Service.
References• Norcini, J.J. and Guille, R.A. (2002). Combining tests and setting standards. In
Norman, G., van der Vleuten, C., and Newble, D. (Eds.): International Handbook of Research in Medical Education (pp. 811-834). Dordrecht: Kluwer Press.
• Norcini, J. J. (2003). Setting standards on educational tests. Medical Education, 37, 464-469.
• Norcini, J. J. & Shea, J. A. (1997). The credibility and comparability of standards. Applied Measurement in Education, 10, 39-59.
• Zeiky, M. J. (2001). So much has changed. How the setting of cutscores has evolved since the 1980s. In G.J.Cizek (Ed.), Setting Performance Standards: Concepts, Methods, and Perspectives (pp. 19-52). Mahwah, NJ: Lawrence Erlbaum Associates.