Fellowship Grassroots Session · • Stakeholder panel for Q&A • SPIN update – Rich Mink ......
Transcript of Fellowship Grassroots Session · • Stakeholder panel for Q&A • SPIN update – Rich Mink ......
Fellowship Grassroots SessionAPPD Fall Meeting
Thursday, September 26, 2019
Agenda
• Community building• Stakeholder panel for Q&A• SPIN update – Rich Mink• Brief discussion re: mentoring• Skill building: assessment
Stand up if…
• You are a:• FPD• F-APD• Coordinator• Vice Chair
• Your specialty is…
Name one word….
• To describe what is valuable or enriching about being a FPD/F-APD/coordinator/Vice-chair
• To describe what is most challenging about your role
Stakeholder Q&A
• ABP – Suzanne Woods, MD – Executive VP• ACGME – Caroline Fischer – Executive Director for RCs and Susie
Buchter, MD – Peds RC chair• COPS – Debra Boyer, MD – Executive Committee Chair
SPIN Update
Mentoring Discussion
MSF
Clinical Practice ITE
ScholarlyProduct
Checklist
Milestone
OSCE
Individual Fellow Assessment:From Programmatic Design
to Individual OutcomesAmanda R. Emke, MD, MHPE
APPD, Fall 2019Disclosures: Grant Support [COMSEP]
Session Goals and Objectives
To develop a workplace-based
assessment tool that fits into a competency-
based program of assessment and meets
the needs of all stakeholders.
Objectives• Define a competency-based
program of assessment• Compare and contrast current
frameworks in medical education
• Discuss the advantages and limitations of direct observation
• Review tools for capturing assessment data based on direct observations
An Important Distinction
MSF
Clinical Practice ITE
ScholarlyProduct
Checklist
MilestoneOSCE
= Program Evaluation
An Important Distinction
MSF
Clinical Practice ITE
ScholarlyProduct
Checklist
MilestoneOSCE
= Assessment
Perfo
rman
ce
Time Spent Learning
Goal of Fellowship Training
Assessment
What is a Competency Based Program of Assessment?
Individual Assessment
Data
Expert Judgement
Decision that Competency is
Attained allowing for
Promotion or Advancement
Reflections
360
Checklist
Sim
Clinical Practice Review
ITEWorkplace
Individual Assessment
Data
Expert Judgement
Decision that Competency is
Attained allowing for
Promotion or Advancement
Reflections
360
Checklist
Sim
Clinical Practice Review
ITEWorkplace
What is a Competency Based Program of Assessment?
Individual Assessment
Data
Expert Judgement
High-Stakes
Decision
Reflections
360
Checklist
Sim
Clinical Practice Review
ITEWorkplace
What is a Competency Based Program of Assessment?
Creating a Competency Based Program of Assessment for Your Fellowship
Year 1
Program Objectives instructed upon and assessed in this Year of fellowship training
Creating a Competency Based Program of Assessment for Your Fellowship
Year 1
Program Objectives instructed upon and assessed in this Year of fellowship trainingLearning activities used to instruct on the Program Objectives
Creating a Competency Based Program of Assessment for Your Fellowship
Year 1
More to Come!
Program Objectives instructed upon and assessed in this Year of fellowship trainingLearning activities used to instruct on the Program ObjectivesAssessments
Program Objectives instructed upon and assessed in this Year of fellowship training
Creating a Competency Based Program of Assessment for Your Fellowship
Year 1
Learning activities used to instruct on the Program ObjectivesAssessmentsStructured, documented learner reflection & projectionRequired meeting between learner and coach
Rotation
Program Objectives instructed upon and assessed in this Year of fellowship training
Creating a Competency Based Program of Assessment for Your Fellowship
Year 1
Learning activities used to instruct on the Program ObjectivesAssessmentsStructured, documented learner reflection & projectionRequired meeting between learner and coach
Rotation
Creating a Competency Based Program of Assessment for Your Fellowship
Year 1
Individual Assessment
Data
Rotation
Creating a Competency Based Program of Assessment for Your Fellowship
Year 1
Individual Assessment
Data
DecisionExpert Judgement
+
What are your Program Objectives?
Do you have specific learning activities?
What assessments do you currently employ?
Is everything arranged in a programmatic way?
First Breakout
Assessment Strategies
General Challenges to Workplace Based Assessment
• Lack of a unifying framework
• Shift to competency assessment
• Complexity of assessment task
• Assessors as sources of error
3 Frameworks for Workplace-Based Assessments
Entrustable Professional Activity
Competency
Competency
Competency
Describes progress along a trajectory.
Divides competence into domains.
Combines domains into tasks.
Physician
Competency
Competency
Competency
Competency
Competency
Competency
Developmental Analytic Synthetic
AMEE Guide No 78. Medical Teacher. 2013
Comparing the ThreeASSUMPTIONS DISADVANTAGES
• Each stage supersedes prior
• Domains progress differently• Understanding true difference
between levels
ADVANTAGES
• Demonstrates individual progress
EXAMPLE
Dreyfus
AMEE Guide No 78. Medical Teacher. 2013
Comparing the ThreeASSUMPTIONS DISADVANTAGES
• Each stage supersedes prior
• Domains progress differently• Understanding true difference
between levels
ADVANTAGES
• Demonstrates individual progress
• Combining discrete elements equals competence
• Elements can be measured discretely
• Covers all aspects of being a “good” physician
• Allows assessment & feedback on specifics
• Extensive descriptions not easily comprehended by clinicians
• Weak connection with clinical activities
EXAMPLE
ACGME
AMEE Guide No 78. Medical Teacher. 2013
Comparing the ThreeASSUMPTIONS DISADVANTAGES
• Each stage supersedes prior
• Domains progress differently• Understanding true difference
between levels
ADVANTAGES
• Demonstrates individual progress
• Combining discrete elements equals competence
• Elements can be measured discretely
• Covers all aspects of being a “good” physician
• Allows assessment & feedback on specifics
• Extensive descriptions not easily comprehended by clinicians
• Weak connection with clinical activities
• Clinical tasks are complex, social, & require multiple domains applied simultaneously
• Strong connection with true clinical work
• Highly authentic
• Specific reasons for or against progression may not be identified
EXAMPLE
EPAs
AMEE Guide No 78. Medical Teacher. 2013
The Chocolate Experiment
Second Breakout
Really a combination…
OR
Addressing the Complexity and Limitations of Competency Assessments in the Workplace Based
Addressing the Complexity and Limitations of Competency Assessments in the Workplace Based
InferenceObservation
Objectivity Subjectivity
Inference Observation
PC1 – Provide transfers of care that ensure seamless transitions
PC2 – Make informed diagnostic and therapeutic decisions that result in optimal clinical judgement
MK1 – Locate, appraise, and assimilate evidence from scientific studies related to patient care
PBLI3 – Use information technology to optimize learning and care delivery
PROF4 – The capacity to accept that ambiguity is part of clinical medicine
Objectivity Subjectivity
PC1 – Provide transfers of care that ensure seamless transitions
PC2 – Make informed diagnostic and therapeutic decisions that result in optimal clinical judgement
MK1 – Locate, appraise, and assimilate evidence from scientific studies related to patient care
PBLI3 – Use information technology to optimize learning and care delivery
PROF4 – The capacity to accept that ambiguity is part of clinical medicine
Objectivity Subjectivity
Bias
• Leniency or Generosity• Recency• Spillover• Central tendency• Halo or Confirmatory• Unconscious
Bias
• Broad, systematic sampling of clinical situations• Short rating instruments• Immediate feedback for teaching and learning • Maintenance of written performance notes support
delayed ratings• Feedback to raters about their ratings• Use formal and informal observations• Promotion decisions made via group review• Establish the meaning of scale items or scores
Mitigating Bias
Williams et al. Teach Learn Med. 2003.
Examples of Workplace Based Assessment Tools
• Mini-Clinical Evaluation Exercise (mini-CEX)• Clinical Encounter Cards• Clinical Work Sampling• Blinded Patient Encounters• Direct Observation of Procedural Skills• Case-Based Discussion/Chart Stimulated Recall• Multi-Source Feedback
Examples of Workplace Based Assessment Tools:Clinical Encounter Cards & Clinical Work Sampling
Purpose Rate performance based on direct observation of patient encounterRating Scale Based on expected level of performanceCapture Mechanism 4x6 card (traditionally)
Electronic platformSupporting Evidence Feasible, valid, reliable
(minimum 8 encounters)Unique Components to the Assessment
Development & commitment to an action plan
• Analytic Framework would rate each skill separately
• Synthetic Framework would rate global encounter
Examples of Workplace Based Assessment Tools:Clinical Encounter Cards & Clinical Work Sampling
• Trainee Name• Observer Name• Date• Clinical Location or Context• General Problem
• Airway/Breathing• CVS/Circulation• GI• Neuro• Psych/Behavior
• Patient Complexity or Acuity• Written Comments
• Strengths• Suggestions for Improvement• Agreed Actions or Learning Plan
• Performance • unsatisfactory • below expectations • at expectations • above expectations • ready for advancement/independence
• Entrustment • observation only • direct observation • indirect observation • unobserved • supervises others
Examples of Workplace Based Assessment Tools:Direct Observation of Procedural Skills
Purpose Assessment of procedural skillsRating Scale Based on expected level of performanceCapture Mechanism Paper or electronicSupporting Evidence Differs based on toolUnique Components to the Assessment
• Documents number of times assessor has observed any trainee• Documents number of times trainee has performed this procedure• Assesses individual steps/components of procedure PLUS overall
ability to perform procedure
3 with good validity evidence1. Operative Performance Rating Scale2. Mayo Clinical Skills Assessment Test3. Ottawa Surgical Competency Operating Room Evaluation
Expected time commitment is 20 minutes – 15 minutes of observation time & 5 minutes for
feedback at completion of procedure
Examples of Workplace Based Assessment Tools:Case-Based Discussion & Chart Stimulated Recall
Purpose Varies but should focus on only oneRating Scale Based on expected level of performanceCapture Mechanism Paper or electronicSupporting Evidence Good triangulation with other assessments of clinical reasoningUnique Components to the Assessment
• Multiple experiences with different faculty• Very useful ”diagnosing” struggling learners• Discussion focuses on only one area of competency but form can
collect data on all of them
Expected time commitment is 25 minutes – 20 minutes of observation time & 5 minutes for
feedback at completion of procedure
• Clinical Reasoning• Referral or Consultation Requests• Follow Up & Future Planning• Professionalism• Documentation
Examples of Workplace Based Assessment Tools:Case-Based Discussion & Chart Stimulated Recall
Purpose Varies but should focus on only oneRating Scale Based on expected level of performanceCapture Mechanism Paper or electronicSupporting Evidence Good triangulation with other assessments of clinical reasoningUnique Components to the Assessment
• Multiple experiences with different faculty• Very useful ”diagnosing” struggling learners• Discussion focuses on only one area of competency but form can
collect data on all of them
Expected time commitment is 25 minutes – 20 minutes of observation time & 5 minutes for
feedback at completion of procedure
• Clinical Reasoning• Referral or Consultation Requests• Follow Up & Future Planning• Professionalism• Documentation
Considerations When Developing a Workplace Based Assessment
Framework
Context
Rater
Tool
Bias
• Should the same assessments be conducted everywhere?• Should the same competencies, skills, knowledge, attitudes be captured in all locations?• What are the facilitators and barriers to capturing the assessments?
• What facets of competence can be observed or should be taken into account?• How do my faculty currently talk about superior, adequate, and unacceptable performance?
• Who is the right person for the job?• How does that person receive appropriate training?
• How can I promote co-production?
• How can I minimize bias?
References
• Louis Pangaro & Olle ten Cate (2013) Frameworks for learner assessment in medicine: AMEE Guide No. 78, Medical Teacher, 35:6, e1197-e1210, DOI: 10.3109/0142159X.2013.788789
• Reed G. Williams , Debra A. Klamen & William C. McGaghie (2003) SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings, Teaching and Learning in Medicine, 15:4, 270-292, DOI: 10.1207/S15328015TLM1504_11
• Holmboe, E. S., & Hawkins, R. E. (2018). Practical guide to the evaluation of clinical competence. 2nd Edition. Philadelphia, PA: Mosby/Elsevier.