Judging Clinical Competence - KUMC · American Board of Anesthesiology, BOI. February 2012 ! AQI...
8
1 Judging Clinical Competence Robert S. Lagasse, MD Professor & Vice Chair Quality Management & Regulatory Affairs Department of Anesthesiology Yale School of Medicine New Haven, CT 64 th Annual Postgraduate Symposium on Anesthesiology April 11, 2014, 15:45-16:30, InterContinental Kansas City at the Plaza, Kansas City, MO Disclosure Dr. Lagasse has had no relevant financial relationship with any commercial entity related to the content of this lecture. Dr. Lagasse has no potential conflict of interest related to the content of this lecture – ASA Representative to the Joint Commissions Professional Technical Advisory Committee until December 2013 – Member of Steering Committee for the CDC/ CMS Surgical Care Improvement Project (SCIP) Objectives After attending this lecture, participants will be able to: 1. Identify the methods of judging the clinical competence of anesthesiologists; 2. Define the limitations of physician level performance measures for judging the competence of anesthesiologists 3. Predict future trends in Maintenance of Certification for judging competence Judging Physician Competence National Practitioners’ Data Bank – State Professional Review Board – Licensure Peer Review – Structured Peer Review – Vitez Model, Lagasse Model Maintenance of Certification in Anesthesiology I Professional Standing II Lifelong Learning & Self-assessment III Cognitive Examination IV Practice Performance Assessment & Improvement National Practitioners’ Data Bank Payments made on behalf of physicians in connection with medical liability (1986) Sanctions against licenses, clinical privileges, and professional societies’ membership privileges (2010) No denominator data to calculate rate of medical malpractice closed claims, just the raw number Health Care Quality Improvement Act of 1986 (42 U.S.C. 11101) Malpractice Litigation & Human Errors 37,924 anesthetics performed (1992-94) 13 cases in which human error, as judged by peer review, led to disabling injury 18 cases involving legal action No relationship between malpractice litigation and human errors National Practitioners’ Data Bank lacks face validity as a measure of competence Edbril and Lagasse. ANESTHESIOLOGY 91:848-855, 1999
Transcript of Judging Clinical Competence - KUMC · American Board of Anesthesiology, BOI. February 2012 ! AQI...
1
Judging Clinical Competence
Robert S. Lagasse, MD Professor & Vice Chair Quality Management & Regulatory Affairs Department of Anesthesiology Yale School of Medicine New Haven, CT
64th Annual Postgraduate Symposium on Anesthesiology April 11, 2014, 15:45-16:30, InterContinental Kansas City at the Plaza, Kansas City, MO
Disclosure
l Dr. Lagasse has had no relevant financial relationship with any commercial entity related to the content of this lecture.
l Dr. Lagasse has no potential conflict of interest related to the content of this lecture – ASA Representative to the Joint
Commissions Professional Technical Advisory Committee until December 2013
– Member of Steering Committee for the CDC/CMS Surgical Care Improvement Project (SCIP)
Objectives
After attending this lecture, participants will be able to:
1. Identify the methods of judging the clinical competence of anesthesiologists;
2. Define the limitations of physician level performance measures for judging the competence of anesthesiologists
3. Predict future trends in Maintenance of Certification for judging competence
Judging Physician Competence
l National Practitioners’ Data Bank – State Professional Review Board – Licensure
l Peer Review – Structured Peer Review – Vitez Model, Lagasse Model
l Maintenance of Certification in Anesthesiology I Professional Standing II Lifelong Learning & Self-assessment III Cognitive Examination IV Practice Performance Assessment & Improvement
National Practitioners’ Data Bank
l Payments made on behalf of physicians in connection with medical liability (1986)
l Sanctions against licenses, clinical privileges, and professional societies’ membership privileges (2010)
l No denominator data to calculate rate of medical malpractice closed claims, just the raw number
Health Care Quality Improvement Act of 1986 (42 U.S.C. 11101)
Malpractice Litigation & Human Errors
l 37,924 anesthetics performed (1992-94) l 13 cases in which human error, as judged
by peer review, led to disabling injury l 18 cases involving legal action l No relationship between malpractice
litigation and human errors l National Practitioners’ Data Bank lacks
face validity as a measure of competence
Edbril and Lagasse. ANESTHESIOLOGY 91:848-855, 1999
2
Peer Review
l Las Vegas Model – Vitez (1990) l Endorsed by ASA for judging competence
– Competence is a human decision; – Best indication of competence is
outcome; and – Humans are inherently fallible
l Included system factors – Lagasse (1993)
Vitez T. J Clinical Anesthesia 1990; 2: 280-287
Error Analysis: Human Factors
Improper technique Equipment misuse / operator error Disregard of available data Failure to seek appropriate data Inadequate knowledge Supervision of residents Communication error Lack of professionalism
Lagasse et al. Anesthesiology 82: 1181-8, 1995
Technical accidents Equipment failure Limitation of therapeutic standards Limitation of diagnostic standards Limitation of resources available Limitation of supervision Failure of communication Lack of professionalism
Error Analysis: System Factors
Lagasse et al. Anesthesiology 82: 1181-8, 1995
Distribution of Contributing Factors
Lagasse et al. Anesthesiology 82: 1181-8, 1995
System Factors (92.2%) Technical accidents Limited therapeutic standards Limited diagnostic standards Limited supervision
Human Factors (7.8%) Improper technique Failure to seek appropriate data Disregard of available data Inadequate knowledge
Face Validity
“I should estimate that in my experience most troubles and most possibilities for improvement add up to proportions something like this: 94% belong to the system (system errors are the responsibility of management) 6% special (human errors are the worker’s responsibility)”
W. Edwards Deming
Deming WE. Out of the Crisis. MIT, Boston 1986
Point 8: Drive out fear...
MMC
60 65 70 75 80 85 90 95
100
% S
elf-
repo
rted
0 1 Month
4 2 3 5 6 7 8 9 10 11 12
50
YNHH
3
Point 8: Drive out fear...
Providers fear that reporting human errors: l Increases risk of malpractice litigation l Suggests that they are less
competent than their colleagues
Human Error Rates & Competence
l All adverse perioperative outcomes between January 1, 1998 and December 31, 2008 were reviewed
l 323,879 anesthetics administered l 104 adverse events attributed to human
error by the anesthesia provider l 3.2 human errors per 10,000 anesthetics
Akerman & Lagasse. ASA Annual Meeting A386, 2010
Survey of Significant Human Error Rates
l Human error rates indicative of the need for remedial training 10 per 10,000 anesthetics
l Human error rates suggestive of incompetence 12.5 per 10,000 anesthetics
Akerman & Lagasse. ASA Annual Meeting A386, 2010
Sample Size, Alpha & Power (1- β)
Power (1-fraction of incompetent providers judged competent)
Alp
ha (f
ract
ion
of
com
pete
nt p
rovi
ders
ju
dged
inco
mpe
tent
)
Akerman & Lagasse. ASA Annual Meeting A386, 2010 Sa
mpl
e Si
ze
Power of Peer Review
If we were willing to be wrong about 1 out of 100 anesthesiologists judged to be incompetent (alpha error 0.01) and 1 out 20 anesthesiologists judged to be competent (beta error 0.05), then sample sizes of 21,600 anesthetics per anesthesiologist would be required.
Akerman & Lagasse. ASA Annual Meeting A386, 2010
ASA PS as Indicator of Perioperative Risk
Saubermann & Lagasse. Mount Sinai J Med. 79:46-55, 2012
ASA PS predicts:
-Outcome Rate
-Outcome Severity
-Nonlinear; interactive complexity
4
ASA PS as Indicator of Human Error Rate
ASA PS predicts:
-Human Error Rate
-Nonlinear; interactive complexity
- Denominator should be judgments, not cases
0
0.002
0.004
0.006
0.008
0.01
0.012
0.014
0.016
0.018
1 2 3 4 5 ASA Physical Status
Human Error Rate (per 1000 cases)
Maintenance of Certification in Anesthesiology (MOCA)
I Professional Standing II Lifelong Learning & Self-Assessment III Cognitive Examination IV Practice Performance Assessment &
Improvement *
American Board of Anesthesiology, BOI. February 2012
Maintenance of Certification in Anesthesiology (MOCA)
American Board of Anesthesiology, BOI. February 2012
Diplomates Certified or Recertified in 2010 - 2014
Professional Standing
l 10th Amendment authorizes laws to protect health, safety and welfare of citizens
l State Medical Boards license MDs l Initial licensure is relatively rigorous
– Medical school, postgraduate training, background checks – USMLE (3 step process)
l Renewal process is less rigorous (no exam) – NPDB review, unrestricted practice, no disabilities, CME
American Board of Anesthesiology, BOI. February 2012
Professional Standing USMLE
l Step 1 – multiple choice exam – Assesses knowledge and application of the basic sciences,
including scientific principles for lifelong learning
l Step 2 – multiple choice exam & patient models – Assesses clinical knowledge and skills essential for the
provision of safe and competent patient care under supervision – Clinical skills assessed include information gathering, physical
examination, and communication
l Step 3 – multiple choice exam – Emphasis on unsupervised ambulatory patient management
American Board of Anesthesiology, BOI. February 2012
Professional Standing Renewal Process
l No examination l National Practitioners’ Data Bank review l Unrestricted practice l No physical or mental disabilities l Continuing Medical Education
American Board of Anesthesiology, BOI. February 2012
5
Lifelong Learning & Self-Assessment
l 350 CME/10 year cycle (>250 Category 1) – < 70 CME per calendar year – SEE Program or ACE Program (> 60 CME) – ASA or ABMS Patient Safety Programs
(> 20 CME) l Monitoring is not rigorous
American Board of Anesthesiology, BOI. February 2012
Lifelong Learning & Self-Assessment
2009 Cochrane Collaboration l Educational meetings, alone or combined with
other interventions, can improve professional practice and patient outcomes
l mixed interactive & didactic formats, and focusing on outcomes perceived as serious, may increase effectiveness
l Not likely to be effective for changing complex behaviors
Forsetlund L et al., Cochrane Database of Systematic Reviews, 2009
Cognitive Examination
l 200 multiple choice questions – 75% general topics – 25% pediatric, cardiothoracic, and obstetric
anesthesia, along with critical care and pain medicine
– >90% pass rate per exam l Unlimited attempts permissible (8X)
– No earlier than the seventh year of their 10-year MOCA cycle
– Offered twice per year
American Board of Anesthesiology, BOI. February 2012
Cognitive Examination
l Unlimited attempts (8 times) – No earlier than the seventh year of
their 10-year MOCA cycle – Offered twice per year
l >90% pass rate per exam
American Board of Anesthesiology, BOI. February 2012
Practice Performance Assessment & Improvement
American Board of Anesthesiology, BOI. February 2012
Attestation: The ABA solicits references to verify clinical activity and participation in practice improvement activities Case Evaluation: 4-step process to assess practice and implement changes to improve Simulation Education Course: A contextual learning opportunity to assess and improve in areas such as crisis management
Attestation
American Board of Anesthesiology, BOI. February 2012
l Due in year 9 of the 10-year cycle l Clinical activity information
– Primary practice type (e.g., anesthesiology, critical care medicine, pain medicine, etc.)
l Contact information for three references – Institution Based – Chief of Anesthesia, Practice Group
President, Medical Director, etc. l Supervisory roles – not a peer review
– Office Based – 3 physicians who refer to your practice
6
Case Evaluation
American Board of Anesthesiology, BOI. February 2012
1. Collect outcome data or patient feedback 2. Compare data with guidelines, expert
consensus, or peer data 3. Design and implement a plan to improve
outcomes using clinical reminders, education, system/process changes, or clinical pathways
4. Collect new data with goal to improve or maintain a high standard of practice
Case Evaluation
American Board of Anesthesiology, BOI. February 2012
l May be done by a group or by an individual l If group approach used, it must be possible
to extract the individual diplomate’s data l Sample case evaluations on ABA Web site
– Nausea and Vomiting – Surgical Site Infections – Hypothermia – Perioperative Beta Adrenergic Blockade
MOCA Case Data Reintubation Rate
A1632, ASA Annual Meeting, Orlando, 2008
REINTUBATIONS
0.00 0.20 0.40 0.60 0.80 1.00 1.20 1.40
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 Year
Rei
ntub
atio
n R
ate/
1000
cas
es
MOCA Case Evaluation Reintubation Rate
Individual Provider Reintubation Rates
0
2
4
6
8
10
12
1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
Year
Rei
ntub
atio
n pe
r 1,0
00 c
ases
A1632, ASA Annual Meeting, Orlando, 2008
ASA AQI Case Evaluations
American Board of Anesthesiology, BOI. February 2012
l Obstructive Sleep Apnea l Perioperative Hyperglycemia l Mask Ventilation l Massive Transfusion Therapy l Prevention and Management of Local
Anesthetic Systemic Toxicity l Postoperative Epidural Catheter
Management During LMWH Administration
AQI Case Evaluation
American Board of Anesthesiology, BOI. February 2012
l AQI PPAI courses are designed as three stage performance improvement activities Stage 1: Audit, Educate, Compare (5 CME) Stage 2: Design, Execute a Performance Improvement Plan (5 CME) Stage 3: Re-Audit, Compare, Reflect (10 CME)
l Cost: Member $220 & Non-member $290
7
Simulation
American Board of Anesthesiology, BOI. February 2012
l ASA-endorsed simulation center l May be completed in a subspecialty l A contextual learning opportunity in
areas such as crisis management (not a knowledge or skills assessment)
l Improves performance in simulators l Cost: $1,800 per person
Simulation
A Byrne & J Greaves. Br J of Anaesthesia. 86(3):445-50 (2001)
l Simulators can generate a variety of tasks that can be used as the basis for the performance assessment
l Simulators can be used to measure adherence to protocols
l Scoring systems in response to simulated situations appear to show good inter-rater reliability
Simulation
A Byrne & J Greaves. Br J of Anaesthesia. 86(3):445-50 (2001)
l Within-subject and within-group variability calls into question stimulus-response expectations of the investigators
l Few studies to date have specially designed assessment to address the questions of validity and reliability
Summary: Judging Competence
Through MOCA
l Initial licensure is rigorous, but renewal does not involve an examination
l CME has a small benefit; not behavioral l Multiple choice exam; multiple attempts l Case evaluation represents a small
individual sample in highly variable practice l Simulation does not include assessment
Age-related Decline in Competency
l Legal Implications – Age Discrimination in Employment Act of 1967 – Exemptions: 1) good cause, and 2) age is a Bona
Fide Occupational Qualification (BFOQ)
l BFOQ burden of proof 1) Reasonable to believe that all or most employees
of a certain age cannot perform the job safely, or 2) It is impossible or highly impractical to test
employees' abilities to tackle all tasks associated with the job on an individualized basis
Recovery from Substance Abuse
l No examination, peer review or simulation training for a recovering physician that could establish if, or when, it is safe to return to work
l Despite mandatory surveillance, there is a high rate of recidivism
l Competence assessment complicated by potential for rapid change and high stakes
8
Policing Our Own
l Physicians self-determine competency to care for patients, when they should retire, and when it is safe to return to work after recovering from substance abuse.
l We need peer assessment of risk-adjusted indicators with risk improved statistical power, frequent written examination, regular simulation assessments, and mandatory retirement
