ACGME 2016 Annual Education Conference SES052 How to...
Transcript of ACGME 2016 Annual Education Conference SES052 How to...
![Page 1: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/1.jpg)
ACGME
2016 Annual Education Conference
SES052
How to Use Milestones to Provide an Evidence-based
Early Warning System for Identifying the
Academically Vulnerable Resident
February 26, 2016
Ann Dohn, MA, DIO and Nancy Piro, PhD
![Page 2: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/2.jpg)
Conflict of Interest
• Ann Dohn, MA - No Conflicts of Interest to Declare
• Nancy Piro, PhD - No Conflicts of Interest to Declare
![Page 3: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/3.jpg)
Agenda - Criticality of Early Identification
• Results of a national survey of DIOs and program directors on Milestones
• Milestones as an early identification of resident academic vulnerability – Evaluations, Safety (SAFE) reports, in service exam scores– Interview behaviors, letters of reference, and summative evaluations
from prior training programs.
• Mining the Milestone Data and Other Data to Identify at Risk Residents– Making it easy
• Tools which summarize resident Milestones as well as other data to facilitate collaborative discussion and remediation planning
• Using learning resources including educational testing and consultation with appropriate specialists
• Electronic resource toolkit to “Take Home”
![Page 4: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/4.jpg)
Results of a 2015 National Survey of Program
Directors on Milestones
5.34%
24.43%
40.46%
18.32%
9.16%
2.29%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Very StronglyAgree
Strong Agree
Agree
Disagree
Strongly Disagree
Very StronglyDisagree
“Milestone evaluations increase our (PDs) documentation of traineeweaknesses/areas for improvement”
![Page 5: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/5.jpg)
Results of a 2015 National Survey of DIOs on Milestones
33.33%
25%
41.67%
“Milestone evaluations increase our (DIOs) documentation of trainee weaknesses/areasfor improvement”
All Agreed
![Page 6: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/6.jpg)
This would never happen…
“What do you mean I have to put my resident on probation? I want to terminate them at the end of the month!”
![Page 7: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/7.jpg)
Evaluations
• Competency based faculty evaluations of residents (rotations)
• 360 evaluations (Techs, Allied health Professionals, nurses, patients, families, coordinators, etc)
![Page 8: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/8.jpg)
‘Data Mining’ Beyond Evaluations - Other Data Sources
• SAFE reports
• In service exam scores
• Interview, meeting and other observed behaviors
• Letters of reference
• Summative evaluations from prior training programs
– Awareness of sources of potential evaluator bias
![Page 9: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/9.jpg)
Leveraging Resident Management System
(RMS)Tools - if Available
• RMSs – becoming more feature rich
– CCC Online Functionality – Assign Files, Upload all required data for Preliminary Review, etc
– Curriculum
• Goals and Objectives and learning outcomes by rotation
• Teaching and Assessment methodologies
– Evaluation tool development
• Sharing between programs and institutions
![Page 10: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/10.jpg)
Leveraging Resident Management System
(RMS)Tools - if Available
• Conference attendance statistics
– Core competencies linked to specified conferences
– Attaching conference materials for later reference
• Procedures and levels; linked procedure evaluations
![Page 11: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/11.jpg)
• Resident portfolio toolsQI participation and outcomesScholarly Activity logs
Leveraging Resident Management System
(RMS)Tools, if Available
![Page 12: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/12.jpg)
Leveraging Your Program Coordinator
![Page 13: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/13.jpg)
Evaluations
Milestone End of Rotation (Total)
Medical Knowledge
Patient Care
Interpersonal & Communication Skills
Professionalism
Systems-Based Practice
Practice-Based Learning & Improvement
Creating a Resident Performance ProfileStep 1 – Defining what to track (Example from Program)
![Page 14: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/14.jpg)
In-service Assessments (MK; PC)
Routine procedure technical skills
assessment: Level 2-3
Complex procedure technical skills
assessment: Level 3-4
Medical Knowledge Assessments
Creating a Resident Performance ProfileStep 1 – Defining what to track
![Page 15: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/15.jpg)
Case Logs / Clinical Experience (PC)
VAGINAL DELIVERY
CAESAREAN SECTION
PEDIATRICS
PEDIATRICS UNDER 3
CARDIAC
ENDOVASCULAR
Creating a Resident Performance Profile Step 1 – Defining what to track
![Page 16: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/16.jpg)
Quality Improvement/Patient Safety
QI Project
QI Committee Participation
QI Course Work (e.g., IHI)
SAFE Report / Adverse Event Review
Patient Handover Evaluations
Creating a Resident Performance ProfileStep 1 – Defining what to track
![Page 17: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/17.jpg)
Practice-Based Learning
Patient outcomes / Case Study Presentation
Scholarly Activity: Research study
Scholarly Activity: PublicationsPresentation at Internal and National Meeting
Creating a Resident Performance ProfileStep 1 – Defining what to track
![Page 18: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/18.jpg)
Communication
Patient Feedback
Staff Evaluation
Medical Student Feedback
Patient Handover Evaluations
Creating a Resident Performance ProfileStep 1 – Defining what to track
![Page 19: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/19.jpg)
Patient Communication Data – Questions and Scale
19
Resident Assessment
“How well did the resident physician…”
Very Poor Poor Fair Good Very Good N/A
1. Greet you in a way that made you feel comfortable? 1 2 3 4 5 N/A
2. Treat you with respect? 1 2 3 4 5 N/A
3. Show interest in your ideas about your health? 1 2 3 4 5 N/A
4. Understand your main health concerns? 1 2 3 4 5 N/A
5. Pay attention to you (look at you, listen carefully)? 1 2 3 4 5 N/A
6. Let you talk without interruptions? 1 2 3 4 5 N/A
7. Give you as much information as you wanted? 1 2 3 4 5 N/A
8. Talk in terms you could understand? 1 2 3 4 5 N/A
9. Check to be sure you understood everything? 1 2 3 4 5 N/A
10. Encourage you to ask questions? 1 2 3 4 5 N/A
11. Involve you in decisions as much as you wanted? 1 2 3 4 5 N/A
12. Discuss next steps, including any follow-up plans? 1 2 3 4 5 N/A
13. Show care and concern? 1 2 3 4 5 N/A
14. Spent the right amount of time with you? 1 2 3 4 5 N/A
Open-ended questions Open Response
15. What did you like about your resident’s communication?
16. How can this resident improve?
17. Do you have any other comments, questions, or concerns?
![Page 20: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/20.jpg)
Creating a Resident Performance Profile
Step 2 - Link data sources to milestones
Evaluations Milestones
Milestone End of Rotation (Total) 24
Medical Knowledge MK A1
Patient Care PC A1-10
Communication ICS A1-2
Professionalism PROF 1-5
Systems-Based Practice SBP A1-2
Practice-Based Learning PBLI A1-4
Quality Improvement/Patient Safety
QI Project PBLI1-2
QI Committee Participation PBLI1-2
QI Course Work (e.g., IHI) PBLI1-2
SAFE Report / Adverse Event Review PROF1/2; SBP1
Patient Handover Evaluations SBP1; ICS1
![Page 21: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/21.jpg)
Milestone Data Translation to Numerical Data
![Page 22: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/22.jpg)
Creating a Resident Performance Profile
Step 3: CCC defines performance ranges
At or Above Expectation:
2.8 and higher
Below Expectation:
1.7 – 2.7
Remediation:
Below 1.7
Example:
For all aggregate milestone evaluation scores for a PGY 3, the CCC defines these ranges:
Hypothetical CCC Defined Ranges:
![Page 23: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/23.jpg)
Creating a Resident Performance Profile
Step 4 – Set conditional formatting
![Page 24: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/24.jpg)
Example: Aggregate milestone evaluation data cells
Highlight cells to apply the conditional formatting
Creating a Resident Performance Profile
Step 4 – Set conditional formatting
![Page 25: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/25.jpg)
Click on “Conditional Formatting”“Highlight Cells Rules”
Creating a Resident Performance Profile
Step 4 – Set conditional formatting
![Page 26: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/26.jpg)
Select, “Greater Than” “Less Than” or “Between” to Set Value RangesChoose the corresponding fill color (e.g., red, yellow, green)
Creating a Resident Performance Profile
Step 4 – Set conditional formatting
![Page 27: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/27.jpg)
Creating a Resident Performance Profile
Compiling and centralizing data
![Page 28: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/28.jpg)
Resident Performance Profile:
Step 5: Enter in data
![Page 29: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/29.jpg)
Creating a Resident Performance Profile
Visual Trends and Detailed Data
![Page 30: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/30.jpg)
When Remediation is Evident
![Page 31: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/31.jpg)
Employing Resident Management System (RMS)Tools - if
Available to Demonstrate Performance in a Peer Context
• Aggregate reporting and graphic summaries• Peer or departmental average, individual average, minimum and
maximum scores, standard deviation or listing of all scores
![Page 32: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/32.jpg)
What To Do with an Underperforming Resident?
1. Look at your tracker for low scores… probe…
![Page 33: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/33.jpg)
What can you do?
1. Engage the resident at the first sign of troubling results
2. Formulate an action plan (talk with your faculty)
Assign a mentor
Are institutional resources needed (DIO?)
3. PD must commit to close monitoring and follow-up
![Page 34: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/34.jpg)
Case Study #1 (Disclaimer-all names and Depts and
Details have been altered)
• SITUATION:
– Neurosurgery resident
– Low in-service exam scores – 1 %
– Unable to keep up with ‘paper work’ and electronic charting - esp. discharge summaries
– Great interpersonal skills
– Outstanding surgical skills
– Former concert musician
• ACTIONS:
• RESULTS
![Page 35: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/35.jpg)
Case Study #2 - (Disclaimer-all names and Depts and
Details have been altered)
• SITUATION:
– Advanced Radiology Fellow
– Nursing complaints about his responsiveness
– Good clinical skills
– Excellent test scores
– Widely divergent evaluations (Faculty vs Nursing)
• ACTIONS:
• RESULTS
![Page 36: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/36.jpg)
Using other resources to assist in remediation
• Educational testing
• Consultation with appropriate specialists
• Culture and Linguistic Specialists
• Well Being Resources
• Mentors
![Page 37: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/37.jpg)
Electronic Toolbox for You!
http://med.stanford.edu/gme/GME_Community.html
• Our Video on Conditional Formatting
• The Practice Excel File that Accompanies the Video (Refer to tab PGY3 for practice): Resident Performance Profile Tool
![Page 38: ACGME 2016 Annual Education Conference SES052 How to …med.stanford.edu/.../2016/SES052_ACGME2016.pdfResults of a 2015 National Survey of Program Directors on Milestones 5.34% 24.43%](https://reader036.fdocuments.net/reader036/viewer/2022071216/6048cdf0642c726aa2339047/html5/thumbnails/38.jpg)
Tools Can Be Downloaded @ www.gme.stanford.edu
GME Community Templates