9/17/2015
1
The Clinical Competency Committee and Milestones
High Value or Busy Work?
Benjamin Kinnear, MDUniversity of Cincinnati College of Medicine
[email protected]: @Midwest_MedPeds
Disclosures
I have no financial or nonfinancial disclosures to make.
Acknowledgement
9/17/2015
2
Learning ObjectivesAfter this presentation you should be able to:
1. Describe the importance of multiple, low‐stakes assessments
2. Discuss how this information can be used by clinical competency committees in advancement, promotion, and Next Accreditation System Reporting decisions
3. Detail the advantages and pitfalls of using entrustment over time as an assessment framework
Overview
1. Vox populi
2. Cincinnati’s assessment system with OPAs
3. Iterative approaches to review of milestones data
4. Current CCC review process
5.49
9.58
What’s the problem?
9.00
8.00
7.00
9/17/2015
3
What if our CCC were baseball scouts?
What is the problem?
Residents CCC
Sir Francis Galton‘Vox Populi’
Nature 75, 450‐451 (7 March 1907) | doi:10.1038/075450a0
9/17/2015
4
Nature 75, 450‐451 (7 March 1907) | doi:10.1038/075450a0
Sir Francis Galton‘Vox Populi’
Sir Francis GaltonThis collective guess was not only better than the actual winner of the contest but also better than the guesses made by cattle experts at the fair.
Nature 75, 450‐451 (7 March 1907) | doi:10.1038/075450a0
1. Each individual member of the crowd must have their own independent source of information
2. Each must make individual decisions and not be swayed by the decisions of those around them
3. There must be a mechanism in place that can collate these diverse opinions
ISBN:0385721706
http://www.all‐about‐psychology.com/the‐wisdom‐of‐crowds.html
9/17/2015
5
Continuity Clinic
CardiologyCritical CareMedicineNephrologyHem/OncNight Float
Sports MedicineEmergency Department
RheumatologyGastroenterologyEndocrinologyGeriatricsNeurology
Infectious Diseases Vulnerable Populations
Meaningful Work‐Based Assessments
Longitudinal Assessment ACGME Milestone Reporting
Automated
Content‐Based Observable Practice Activities (COPAs)1. Initiate basal bolus insulin therapy 2. Manage elevated blood pressure3. Manage COPD
Process‐Based Observable Practice Activities (POPAs)1. Minimize unfamiliar terms during patient encounters.2. Use teach‐back method
Observable Practice Activities (OPAs)What residents DO.
Different for every rotation
Similar for every
rotation
Entrustment and Mapping of Observable Practice Activities for Resident Assessment Eric J. Warm MD, Bradley R. Mathis MD, Justin D. Held MD, Savita Pai MD, Jonathan TolentinoMD, Lauren AshbrookMD, Cheryl Lee MD, David Lee MD, Sharice Wood MD, Carl J. FichtenbaumMD, Ryan Munyon MD, Caroline Mueller MD, JGIM 2014 accepted for publication
9/17/2015
6
Rotation: Digestive Diseases Inpatient Wards PGY‐1
Content‐Based OPAs (selected)
1. Write initial admission orders for gastrointestinal bleeding 6. Initiate enteral and parenteral nutrition 9. Perform paracentesis
Entrustment and Mapping of Observable Practice Activities for Resident Assessment Eric J. Warm MD, Bradley R. Mathis MD, Justin D. Held MD, Savita Pai MD, Jonathan TolentinoMD, Lauren AshbrookMD, Cheryl Lee MD, David Lee MD, Sharice Wood MD, Carl J. FichtenbaumMD, Ryan Munyon MD, Caroline Mueller MD, JGIM 2014 accepted for publication
Rotation: Digestive Diseases Inpatient Wards PGY‐1
Process‐Based OPAs (selected)
5. Demonstrate accurate medication reconciliation 12. Use teach‐back method with patients15. Minimize unnecessary care including tests
Entrustment and Mapping of Observable Practice Activities for Resident Assessment Eric J. Warm MD, Bradley R. Mathis MD, Justin D. Held MD, Savita Pai MD, Jonathan TolentinoMD, Lauren AshbrookMD, Cheryl Lee MD, David Lee MD, Sharice Wood MD, Carl J. FichtenbaumMD, Ryan Munyon MD, Caroline Mueller MD, JGIM 2014 accepted for publication
Rotation: Digestive Diseases Inpatient Wards PGY 2‐4
Content‐Based OPAs (selected)
1. Manage gastrointestinal bleeding 2. Manage pancreatitis 9. Manage complications of immunosuppressive therapy
Entrustment and Mapping of Observable Practice Activities for Resident Assessment Eric J. Warm MD, Bradley R. Mathis MD, Justin D. Held MD, Savita Pai MD, Jonathan TolentinoMD, Lauren AshbrookMD, Cheryl Lee MD, David Lee MD, Sharice Wood MD, Carl J. FichtenbaumMD, Ryan Munyon MD, Caroline Mueller MD, JGIM 2014 accepted for publication
9/17/2015
7
Rotation: Digestive Diseases Inpatient Wards PGY 2‐4
Process‐Based OPAs (selected)
5. Manage the interdisciplinary team12. Teach physical findings for junior members of the health care team13. Stabilize patients with urgent or emergent medical conditions
Entrustment and Mapping of Observable Practice Activities for Resident Assessment Eric J. Warm MD, Bradley R. Mathis MD, Justin D. Held MD, Savita Pai MD, Jonathan TolentinoMD, Lauren AshbrookMD, Cheryl Lee MD, David Lee MD, Sharice Wood MD, Carl J. FichtenbaumMD, Ryan Munyon MD, Caroline Mueller MD, JGIM 2014 accepted for publication
Our Rating Scale: Entrustment
1. Resident not trusted to perform skill even with supervision
2. Resident trusted to perform skill with direct supervision
3. Resident trusted to perform skill with indirect supervision
4. Resident trusted to perform skill independently
5. Resident trusted to perform skill at aspirational level
6. Skill was not observed on this rotation (produces no score)
Consider each skill individually and discriminate between relative strengths
and weaknesses for each resident
9/17/2015
8
• If a resident earns high scores, tell us why
• If you can’t justify, please correct
• Your words are VERY important
• Please be detailed and provide vignettes and illustrations to make your points
Don’t write “needs to read more”.Please be specific. Needs to read more what?
Multisource Evaluations
9/17/2015
9
Observable Practice Activities (OPAs)
Process‐Based (POPA)
Mid/End of Rotation Feedback (formative)Entrustment (1‐5)
Content‐Based (COPA)
Curricular Milestones
Reporting Milestones
End‐of‐trainingEPAs
External ReportingACGME report
Longitudinal Assessment(summative)
Mapping Mapping
Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518
*
*
*
*
*
We got the data!
• Annualized over 3 years each resident receives an average of:
• 75 assessment encounters
– 57% are from attending physicians, and 43% are from peer and allied health
• 3703 milestone assessments
– 81% are from attending physicians, and 19% are from peers and allied health
• 4325 words of narrative assessment
Warm EJ, Mathis BR, Ashbrook L , Held JD, Hellmann M, Kelleher M., Kinnear B, Sall D, Tolentino J, Mueller C, Schauer D, Entrustment of Milestones in an Internal Medicine Residency, currently in process
Our CCC
Mountain of milestone data
9/17/2015
10
What is the purpose of the CCC?
• Accountability to the public
• Ensure quality/usability of assessment tools
• Track learner competency
– Longitudinal feedback
– Reporting to ACGME
• Improve the program
• Improve the faculty
Adapted from Clinical Competency Committees: A Guidebook for Programsfrom the ACGME
Observable Practice Activities (OPAs)
Process‐Based (POPA)
Mid/End of Rotation Feedback (formative)Entrustment (1‐5)
Content‐Based (COPA)
Curricular Milestones
Reporting Milestones
End‐of‐trainingEPAs
External ReportingACGME report
Longitudinal Assessment(summative)
Mapping Mapping
Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518
9/17/2015
11
Approximately 3000
Assessments
169 Assessments
150Assessments
177 Assessments
Reporting Milestones
Resident A Resident B
Resident C
Reporting Milestones
Resident A Resident B
Resident C
9/17/2015
12
What happened?
Resident A
Tougher rotations
All PGY‐1 Rotations
n = 87862
General Medicine Wards
n = 15296
General Medicine Hospitalist
n = 13803
Night Medicine
n = 8217
9/17/2015
13
Cardiology
n = 6952
Gastroenterology
n = 5075
Nephrology
n = 4858
Hematology‐Oncology
n = 6300
Cohort 2 Cohort 3 Cohort 4
9/17/2015
14
PGY‐1 Assessments of Senior Residents
N = 9926
Jul Sep Oct DecNovAug Jan Feb
Lessons learned
• Some people are more trusting than others• Some rotations are more trusting than others• Context effects
– type and amount of direct observation– type of physician (even within specialty)– types of OPAs– personal perspective
• Rotation order effects• Time of year effects• Cohort effects
Controlling for Bias
Want to control for:
• Evaluator
• Service
• Type of evaluation
• Actual question
• PGY level
• Year
• Month within the year
Dan Schauer MD
9/17/2015
15
Predicted Scores (PS)
• Created a regression model that predicts what the score should be on average irrespective of resident
P.S.=β0+β1*Evaluator+β2*Service+β3*Question+β4*Month…..
Dan Schauer MD
Resident A
Blue = residentRed = predicted score
9/17/2015
16
Blue = residentRed = predicted score
Blue = residentPink = trend line
9/17/2015
17
Program Director Review
Clinical Competency Committee
CollectPre‐
reviewReview
Deep dive
Reporting
Feedback
Kinnear, et al, The clinical competency committee in the age of milestones – experience from two internal medicine residencies. manuscript under review at JGME
X
Collect Pre‐review Review Deep diveReporting
Feedback
Brief
Not needed
9/17/2015
18
Y
Collect Pre‐review Review Deep diveReporting
Feedback
Guided by prereviewer
Longer
“Hidden” pitfalls…
•Assessment quantity•Assessment quality
Single Resident all assessments (faculty, peers, allied health) n = 3005
9/17/2015
19
Single resident assessments (faculty only) n = 2023
Hammer Time?
• If evaluations are completed > 14 days after the attending portion of rotation, the attending forfeits entire teaching practice plan payment
– Initially lobbied for 5 days
– 1 grace miss
• Fines collected used to reward best teachers
“Mandatory” monthly pre‐rotation faculty meetings
Reminder emails
“Mandatory” Videos
Pay reduction threat
On Time Completion Percentage
9/17/2015
20
Single resident assessments (faculty only) n = 2023
What is the purpose of the CCC?
• Accountability to the public
• Ensure quality/usability of assessment tools
• Track learner competency
– Longitudinal feedback
– Reporting to ACGME
• Improve the program
• Improve the faculty
Adapted from Clinical Competency Committees: A Guidebook for Programsfrom the ACGME
9/17/2015
21
Take home points
• Large amounts of low‐stakes assessments are optimal
• Data begins a story, narratives complete it
• Shared mental model in CCC is key
• Reassess your process frequently – beware of hidden pitfalls
• Your CCC can do more than just remediate/promote
Questions?
Top Related