Mennin Consulting, 2006 Overview of Assessment ESME 2006 Stewart Mennin, PhD.
-
Upload
esther-oneal -
Category
Documents
-
view
226 -
download
0
Transcript of Mennin Consulting, 2006 Overview of Assessment ESME 2006 Stewart Mennin, PhD.
Mennin Consulting, 2006
Overview of AssessmentESME 2006
Stewart Mennin, PhD
Mennin Consulting, 2006 ESME
Objectives
Make more informed decisions about assessment
Apply information to your teaching Describe the relationship between
assessment & learning Discuss criteria for effective assessment Define & apply basic terminology
Mennin Consulting, 2006 ESME
Objectives
Describe selection, supply & performance methods
Link Miller’s pyramid to assessment methods Match assessment methods to your
outcomes Draft assessment plan (blueprint)for your
teaching
Mennin Consulting, 2006 ESME
GOALS OF ASSESSMENT
Enhance student learning Guide faculty teaching efforts Inform curricular development & revision Fulfill responsibilities to students, institution &
society Contribute to research
Mennin Consulting, 2006 ESME
FEATURES OF ASSESSMENT
Assessment is more a curriculum design problem than a measurement problem Part of learning & can not be separate from it Assessment is about the present, past and future of learning Assessment in not a collection of instruments Assessment should be authentic, grounded in what is real
Mennin Consulting, 2006 ESME
EIGHT CRITERIA: EFFECTIVE ASSESSMENT
1. Articulation of a clear conception of intended learning outcomes
2. Use variety of assessment procedures/methods
3. Close agreement between intended outcomes of instruction, learning tasks, & the assessment procedures/methods
4. Adequate sampling of student performanceGronlund, 2000
Mennin Consulting, 2006 ESME
EIGHT CRITERIA: EFFECTIVE ASSESSMENT
5. Procedures that are equitable for all participants
6. Explicit, specific criteria for use in judging successful performance
7. Timely feedback that emphasizes strengths of performance & focuses attention on specific areas in need of improvement
8. A grading & reporting system consistent with the rest of the learning & assessment process
Gronlund, 2000
Mennin Consulting, 2006 ESME
Key Terminology
Formative – Summative Reliability & Validity Criterion Referenced --- Norm referenced
Progress test Sequential test
Mennin Consulting, 2006 ESME
Assign gradesJudge competenceCertify masteryPromotion
Feedback to student & learnerFind misconceptionsIdentify strengths &
weaknessesChange future teachingDevelop learning plan
TeacherTeacher & learner
Milestones & endFrequent & throughout
Decisions / JudgmentImprovement
Use
Evaluator
Timing
Purpose
SummativeFormative
Assessment As Learning
Mennin Consulting, 2006 ESME
Reliability
Consistency of results Reliability is matter of sampling
– Across content
Mennin Consulting, 2006 ESME
Domain of Interest
Test Sample
Test Sample
Mennin Consulting, 2006 ESME
Assessment of Clinical Assessment of Clinical Competence is a Issue of Competence is a Issue of
SpecificitySpecificity
Competent
Mennin Consulting, 2006 ESME
Reliability
Practical suggestions:– Do not rely on short tests
– Sample broadly (content, in time, examiners, patients)
– Consider efficiency in: selection of test format construction test items
Mennin Consulting, 2006 ESME
Reliability
Adequate reliability requires substantial sampling (therefore: resources, testing time, examiners, patients, etc.)– Be efficient
Mennin Consulting, 2006 ESME
Validity
Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.
Knows
Shows how
Knows how
Does
Pro
fess
iona
l aut
hent
icity
Pro
fess
iona
l aut
hent
icity
Mennin Consulting, 2006 ESME
Validity
Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.
Knows
Shows how
Knows how
Does
Pro
fess
iona
l aut
hent
icity
Pro
fess
iona
l aut
hent
icity
Written, Oral orComputer based assessment
Performance or hands on assessment
Mennin Consulting, 2006 ESME
Valid Inferences
Validity is a single concept with different forms of evidence– Content-related evidence– Criterion-related evidence– Construct-related evidence
Mennin Consulting, 2006 ESME
ASSESSMENT METHODS
All assessment methods have limitations and no one method can assess all types of knowledge & skill
Multiple methods are recommended
Mennin Consulting, 2006 ESME
Some Assessment Options:
Selection Item– Multiple Choice Question– One-Best Answer Questions– Extended-Matching Questions
Supply Items– Free Response Questions– Short Answer Questions– Essay Questions
Performance Examinations– Laboratory Practical Examinations (Identification and Concepts)– Clinical Skills (e.g., OSCE)– Other
Mennin Consulting, 2006 ESME
Selection Methods
MCQ Matching Extended Matching True False Multiple True False
Knows
Knows how
Shows how
Does
Mennin Consulting, 2006 ESME
Knows/Knows how
The stimulus format is more important than the response format
KnowsKnows
Knows how
Shows how
Does
Mennin Consulting, 2006 ESME
How Well Written?Stem
a. Poor
b. Good
Mennin Consulting, 2006 ESME
How Well Written?
a.
Stem
b.
c.
a. Poorb. Good
d.
Mennin Consulting, 2006 ESME
How Well Written?
30-year-old man has loss of pain and temperature sensation from the neck down on the right side of the body and on the left side of the face; partial paralysis of the soft palate, larynx, and pharynx on the left; and ataxia on the left. This syndrome is most likely to result from thrombosis of which of the following arteries?A BasilarB Right posterior inferior cerebellarC Left posterior inferior cerebellarD Right superior cerebellarE Left superior cerebellar
Mennin Consulting, 2006 ESME
Extended Matching
For each quotation, select the most likely author.
1. I never teach my pupils. I only attempt to provide the conditions under which they can learn.
2. All genuine knowledge originates in direct experience.
3. To know one’s ignorance is the best part of knowledge.
Authors on Education:
a. D. J. Boorsten e. Lao-Tzu
b. Camus f. S. Scott Obenshain
c. Albert Einstein g. John Pieper
d. Thomas Harris h. Mao Tse-Tung
Mennin Consulting, 2006 ESME
Extended Matching Answers
Authors on Education:
a. D. J. Boorsten e. Lao-Tzu
b. Camus f. S. Scott Obenshain
c. Albert Einsteing. John Pieper
d. Thomas Harris h. Mao Tse-Tung
1. I never teach my pupils. I only attempt to provide the conditions under which they can learn. (c)
2. All genuine knowledge originates in direct experience. (h)
3. To know one’s ignorance is the best part of knowledge. (e)
Mennin Consulting, 2006 ESME
Supply Methods
MEQ Essay (long & short) Practical Short Answer Written simulations
Knows
Knows how
Shows how
Does
Mennin Consulting, 2006 ESME
Performance Assessment
Procedures – skills lab OSCE (with skills stations) Mini CEX Long Case Logs Portfolio Clinical Notes Observation Other Knows
Knows how
Shows how
Does
Mennin Consulting, 2006 ESME
Communication Skills
Interviewing & Communication Skills
Mennin Consulting, 2006 ESME
Essential Elements of Communication Skills
Open the Discussion Build a Relationship Gather Information Understand the Patient’s Perspective Share Information Reach Agreement Provide Closure
Acad Med 2001;76(4):390-393
O rg an ig ram tite l
C lin ica lS cen ario
ClinicalClinicalProblemProblem
Key Feature Approach
Testing time
Case 1
Case 2
Test usingintegral cases
Case 1
Case 2
Case 3
Case 4
Case 5
Case 6
Case 7
Test usingkey features
Mennin Consulting, 2006 ESME
Shows How: OSCE
Make stations as clinically authentic as possible
Global judgments do well in OSCEs
Content specificity is the problem
Knows
Knows how
Shows how
Does
Mennin Consulting, 2006 ESME
Checklist/Global Rating Scale Comparison
Checklists– Dichotomous scores
(yes/no)– Relatively objective
Record rather than interpret
– Station specific
Rating Scales– Ordinal level measures
(poor, good, excellent)– Relatively subjective– Station independent
Mennin Consulting, 2006 ESME
Does: Practice Assessment
Indirect methods– Chart audit– Practice audits– Prescription data………
Direct methods– Clinical (supervisor) ratings– Undercover standardized patients– Video assessment– Patient reports– Peer reports– Clinical work samples– Mini-CEX………
Knows
Knows how
Shows how
Does
Mennin Consulting, 2006 ESME
Clinical Work Samples
Repeated direct observations of clinical performance in practice using (generic) evaluation forms, completed by any significant observer (clinician, nurse, peer…..)
Illustration
Knows
Knows how
Shows how
Does
Mennin Consulting, 2006 ESME
Mini Clinical Examination (Norcini, 1995)
Short observation during clinical patient contact (10-20 minutes)
Oral evaluation Generic evaluation forms completed Repeated at least 4 times by different
examiners (cf. http://www.abim.org/minicex/)
Knows
Knows how
Shows how
Does
Mennin Consulting, 2006 ESME
NEW SKILLS EMPHASIZED
Learning how to learn Self-assessment Humanization Leadership Team skills
Skills of expression Professional behavior Reflection Permanent education
Mennin Consulting, 2006 ESME
Knows Knows How Shows How DoesWritten Tests Essay A A
MCQ A AWritten Exercises Application Test B A
MEQ B APractical B A
Patient Exercises OSCE B ASPs B ATriple Jump B AMini CEX A BLong Case A B
Observations Direct Observation B AMed Record Review B AOral Examination B APeer Evaluation B ARating Scale B A
A = Most effective B = Less effective
Methods & Knowledge
Mennin Consulting, 2006 ESME
Task
Use the table from effective teaching and match assessment methods to your outcome competencies
Mennin Consulting, 2006 ESME
0
10
20
30
40
50
60
Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Ref. Group
Percentage correct
Progress TestScores on one progress test
Faculty of Medicine Maastricht University
Mennin Consulting, 2006 ESME
Mean correct scores on the Progress Test from a number of Maastricht student cohorts (black lines) and reference group of family physicians in training (blue line)
Progress Test Maastricht Test Results ‘77 - ‘88
Faculty of Medicine Maastricht University
Mennin Consulting, 2006 ESME
Planning: Blueprinting
Competencies Time Teaching methods In class -- out of class
Com
pete
ncie
s
Perform-2 Capable
Perform-1 Competent
SupplySelect
Methods
Clinical Skills
Critical Int
KnowComm
Human
Self Assess
ProfessProfess
Mennin Consulting, 2006 ESME
Conclusions
Assessment is less of a psychometric problem, but rather an educational design problem (i.e. how to use assessment strategically for its educational effects)
Assessment requires careful planning and monitoring To effective with the entire competence pyramid
– a mix of methods that depends on the context; not a single method is best or bad
Cees van der Vleuten
Mennin Consulting, 2006 ESME
Acknowledgements
Cees van der Vleuten - Maastrict Deana Richter – New Mexico Teresita McCarty - New Mexico Nancy Sinclair - New Mexico Regina Petroni Mennin – UNIFESP
Mennin Consulting, 2006 ESME
www.menninconsulting.com
Presentations Available at: