Measuring Outcomes for Residency Graduates

Steven L. Frick, MD

Chairman, Dept. of Orthopaedic Surgery

Director of Medical Education, Nemours Children’s Hospital

Professor and Assistant Dean

University of Central Florida College of Medicine

Orlando, Florida

No financial disclosures.

AAOS Program Committee

POSNA Curriculum Committee

POSNA Residents Review

POSNA Treasurer, Board of Directors

JRGOS Board of Directors

ABOS QWTF

ACGME Milestones Project Workgroup

No competency specifically addressing surgical skills

Macy Foundation Report 2011The Content and Format of Graduate Medical EducationRecommendation III-A: The length of GME should be determined by an individual’s readiness for independent practice- demonstrated by fulfillment of nationally endorsed, specialty-specific standards- rather than tied to a GME program of fixed duration.

“nationally endorsed, specialty-specific standards”

Do we have any of those?

Role for CORD

Optimistic versus Pessimistic “an opportunity in every difficulty”

versus “ a difficulty in every opportunity”

Evidence Based Medicine

Integrating individual clinical expertise with the best external

clinical evidence

Outcome

A final product or end result

A. Flexner - 1910

Medicine can be learned but not taught

Active participation required

Need dedicated educators and students

“get comfortable with uncertainty”

Role of professional education

Provide practitioners the intellectual tools to assess information critically, stay abreast of changing knowledge, adapt to continuous change, and reflect on the larger role and responsibilities of the profession in society.

From Time to Heal by Kenneth Ludmerer

“Is there a core body of knowledge and skills that

the finishing resident should possess prior to

starting practice or fellowship?”

- Richard Gross, MD

Need curriculum

and

competency assessment

Residency fundamentally =Master-Apprentice

William Halsted:Residency Training systemIntroduced in 1889 at Johns

Hopkins based on: a fixed period of time for training, structured educational content, actual experience with patients, escalating responsibility for

patient care during training, and a period of supervised practice after formal training.

Remains the cornerstone of surgical training in North America more than a century later

Competency Based Education

Defined by the outcome of the educational process, not the content

Develop weighted curriculum to teach and assess (Farmer, Gross, Wadey)

Assessing competence focuses on what the learner is able to do

How do you assess competency?

"the state or quality of being capable or competent; skill;

ability."

Miller’s model of competence

Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

Prof

essio

nal a

uthe

ntici

ty

Read, Listen Knows

Shows how

Knows how

Does

Performance or “hands on”

Live Demo; Multimedia

OCAP

Dreyfus Model of Skill Acquisition

Mastery Learning Model

-Bloom 1968

Becoming Expert

“The 10,000 Hour Rule”

About 10 years- dedicated practice

Bill Joy- UNIX, Sun Microsystems; Mozart; The Beatles; Bobby Fischer;Bill Gates

Model of complete clinical careModel of complete clinical care

Engage

Empathize

Educate

EnlistFix It

Find It

Opening

Closing

Culturally Competent Care

The ability to understand and work with patients whose beliefs, values, and histories are significantly different from our own.

• Quality of Outcomes

• Patient-Physician Relationship

• Malpractice Claims

CCC Education

Team Harmon

y & Quality of Work

Life

Error Prevention

Medical/Surgical Team Concerns

Cultural Competence in Health Care

Courtesy of A. White, III, MD

Defining / Teaching/ Modeling Professionalism most important

Drs. Cruess body of workHidden curriculum

Social ContractIndividual Awareness

Teach and Model Professionalism

Surgery- tripartite body of knowledgeFrank Wilson, MD

Preoperative - evaluation, indications, planning

Intraoperative - technical execution Postoperative - immobilization, weight-

bearing, PT All 3 necessary for success

Ortho Surgical Education

Interns - pre and post operative care, framework of ortho fundamentals, closed management of fxs

PGY 2/3 - basic decision-making and psychomotor skills

PGY 4/5 - independent decision-making, subspecialty skills, integrate knowledge

Our Educational philosophy at CMC Not training Stimulus - Reaction vs

Stimulus - Thought - Reaction Create one-on-one master-apprentice

situations Graduated responsibility ALWAYS supervised in highest risk activities

(OR) Have to spend enough time with them to

know

I DON’T KNOW

How do you assess competency?

Charlotte Competency Stages

Stage I - do not know anything cannot do anything, and know it

Stage II – know and can do a lot, but do not recognize what you do not know and cannot do DANGER

Stage III – know and can do a great deal, but realize there is much you do not know

“The beginning of a mountaineer’s career, when energy and enthusiasm outpace experience and judgment, is said to be the most dangerous part.”

Photo by Guillaume Dargaud

Setting Standards

Job of Chair and RPD to set standards of excellence

Graduates of program should meet these standards in all core competencies

Assessing competency

Complete 5 years of orthopaedic surgery program under watchful eye of PD

12 months PGY1 / internship, 48 months orthopaedic surgery

Evaluations and comment by faculty, peer evaluations, portfolio (presentations, courses, outcomes instruments), OITE, operative experience log

Consensus of PD and faculty

ABOS

I believe this individual is capable of the competent independent practice of orthopaedic surgery.

Steven L. Frick, MD

Residency Program Director

An Expert- Knows

Knows WHAT to do Knows HOW to do it Knows WHEN to do it (and when not

to) Knows WHY to do it Knows WHEN to ask for help Knows WHAT we don’t know

Is it possible/desirable to define and measure

competency and then

graduate a resident before 5 years?

A Competency-Based Curriculumin Orthopaedic Surgery:

From Idea to Implementation

Markku Nousiainen, MS, MD, MEd, FRCS(C)

Sunnybrook Health Sciences Centre

University of Toronto

Current challenges in residency training

reduction in work hours

reduced time spent in OR

teaching surgical skills

reduction in wait times

improvement in patient safety

reduced training opportunities for residents

}

Competency-based education

“Training process that results in proven competency in the acquisition & application of skills & knowledge to medical practice that is not simply dependent on the student’s length of training & clinical experiences”

“Much of what is counted does not count, and much of what counts cannot be counted.”

Problems

Toronto experience- 5 years, now all in for first time

Still no defined “curriculum” More resource intensive than

traditional pathway = costs more Current environment of GME= very

dependent on Medicare funding Some predict reduction in Medicare

GME funding under PPACA 2010

How much of residency education is experiential?

Can we list / define everything you need to learn?

Can we transfer knowledge gained from experience without making residents have

the experience?

Duty hours 2003

First ortho class with 80 hour work week- double failure rate on part I ABOS certification exam

Similar result 2011 exam takers Why? Does this exam measure competency? Who do you want – 90%ile or 30%ile?

GME-Decade of Accountability

To patients by residents, faculty Patient safety, Resident safety- RPD To residents by faculty, institution Societal demands for assurance of

competency Safe, Effective, Patient centered,

Timely, Efficient, Equitable (IOM) Increased requirements by oversight

organizations – RPD time Professional, ethical behavior

demanded

NAS- Next Accreditation System

Coming to Ortho July 2013 No more site visits, PIFs every 5

years Annual “Biopsy” of 4 things

– Institutional report – Annual survey of residents and faculty– Case logs– Milestones (q 6 mos reports from

Competency Committee)

Self report every 10 years

Ortho Milestones- 18cover PC and MK

All have 5 levels

By graduation resident should be level 4 (competent) in all

For peds- septic arthritis and SC humerus fracture

Surrogates for knowledge in other areas

Milestone- Peds SCH Fracture

NASwww.acgme-nas.org

Institutional reviews (q 18 mos) Milestones reports (q 6 mos)- form a

competency committee Operative experience database Resident annual survey Faculty annual survey (new) ORTHO JULY 1, 2013

Future of Orthopaedic Residency Education

Change is coming Need to protect experience, in addition

to more rigorous evaluation / oversight More evaluations / structured

experiences Remember importance of graduated

independence Milestones will be modified as we go NAS is on the way- BE AN OPTIMIST!

Thank You