Residents In Trouble

M. Rebecca Hoffman, MD, MSPH

December 8, 2008

Academy for Scholarship in Education

Overview

Scope of the Problem Literature Review Study Design Study Results (in progress)

Scope of the Problem

The goal of Family Medicine Residency

Residents’ backgrounds

No guidance

What does the RRC say?

Must have advising system

Must give regular feedback

BUT… no real guidance regarding what/how

Consequences of RIT

For the individual RIT Psychological stress Financial stress Eventual job security

For the program Adequate coverage of duties Morale Faculty time/energy Dismissal

Consequences…

Other consequences Recruiting Impact on patient care

Literature Review

Not much on the prevalence of RIT Reamy et al (1): 25 year in-depth study of one

family medicine residency program 9.1%

Yao and Wright (2): Survey of Internal Medicine PDs– 6.9%

Williams et al (3): SIU Gen Surgery Residency 22%

Kinds of Problems

Knowledge Deficits Attitudinal Problems Interpersonal Conflict Psychiatric Illness Substance Abuse Family Stress Relationship Disruption

Other Classification

Academic Performance Classroom, test

Clinical Performance Applied knowledge/skills

Professional Behavior Workplace behavior, relationships with other

professionals, etc Others

Difficult to classify

So we can name it…

But what do we do with the information?

Literature reviews: “Need to do better” Core content review, quizzes Counseling Treatment of illnesses Repetition (rotation, year) Probation Dismissal Others

Summary of the problem

We know it’s there, but we aren’t sure exactly how common it is

We aren’t good at predicting who will struggle

Once we identify them, we don’t know what to do with them

RIT Family Medicine Study

Primary Aims

1) Determine the prevalence of RIT in the SIU Family Medicine Residency Programs

2) Describe and categorize the types of troubles encountered in the FCM residency programs

RIT Aims...

3) Identify and describe the kinds of processes the programs employ to address deficiencies with RIT

4) Describe outcomes of the residents who underwent remediation

5) Identify early indicators to predict RIT

RIT Study Design

Chart review/resident record -based

Prevalence over significant timeframe 10 years of entering classes Entering class of 1993-entering of 2003

Why?

De-identified data from all 4 programs To further assure confidentiality No one program singled out

RIT Study

Initially modeled after Reamy et al study (1) after discussion with Dr. Reamy about the process in his residency program

Assistance from Reed Williams, PhD and Nicole Roberts, PhD

Methods

Rosters of all entering residents for the 10 years from all programs

Random ID numbers assigned

Records from each program reviewed in their entirety Mostly paper Some electronic

Methods…

Data collection instrument created based on personal experience, literature, and discussion with Reed Williams and Nicole Roberts

Feedback from academic faculty

Suggestions incorporated as needed

Collected for all residents:

US Medical School Grad (Y/N) USMLE or NBOME or COMLEX scores

Step 1, Step 2, Step 3 (when available) Gender (M/F) ITE Scores

PGY1, PGY2, PGY3 (when available) Did the resident have substantial difficulties?

(Y/N)

For those who struggled

Time frame- when first identified Categorization of the main problem

Academic, Clinical, Professionalism, Other Main performance area involved

See list Breadth of performance areas

Academic, Clinical, Professional, Other All performance areas involved

See list

For those who struggled

What remediation attempted? See list

What was the final resolution? Graduated with concerns Graduated without concerns Voluntarily left Dismissed Finished on probation Not documented

Severity Rating

Compared to other residents with performance problems, this resident’s problems were: Likert scale, 1-5

1 among the least serious 5 among the most serious

Narrative descriptions of overall case, time course, etc

Example of Survey Instrument Survey Monkey for data entry

Data Collection

2 raters

Independently reviewed same 5 charts at first program to establish inter-rater reliability, then 3 charts at each other program

Data Collection…

Then independent record review

Any case the reviewer identified as a RIT was then reviewed by the other reviewer independently, ratings discussed after each case

“Borderline cases” also reviewed by both

Data Analysis

Data entered into Survey Monkey for ease of collection

Analysis: descriptive and basic statistics

When all data complete, analysis using SPSS

Results

96 Residents reviewed (as of 12/4/2008)

US Med School Grads: 69% (66)

Male: 65% (62)

Female: 35% (34)

Prevalence of documented RIT: 38.5% (37)

RIT (N=37)

US graduates: 59.5% (22) Foreign graduates: 40.5% (15)

Male: 70.3% (26) Female: 29.7% (11)

Male, US: 40.5% (15) Male, Non-US: 29.7% (11) Female, US: 18.9% (7) Female, Non-US: 10.8% (4)

RIT

Performance Area Number of Residents

with the problem

Academic 6

Clinical 19

Professional 11

Other 1

Most Important Single Problem Area? Knowledge (10) Putting everything together (3) Data interpretation/diagnosis (2) Data collection (2) Incomplete paperwork/charts (2) Lack of motivation/interest (2) Treatment/management (2)

Breadth of Problems

Area(s) of Problems Number

Academic Only 1

Academic and Clinical 7

Academic and Professional 3

Clinical Only 1

Clinical and Professional 12

Academic, Clinical, and Professional 6

Professional Only 6

Other 1

Breadth of problems

Number of performance areas involved ranged from 1 to 23 Average: 7

See breakdown of individual RIT on handout

Remediation

Ranged from “none” or “told to improve” to 12 or more interventions for a single resident

Most commonly reported: Increased meetings with advisor/mentor

Outcomes of RIT (residency)

26 graduated from the program (70.3%) 11 did not graduate (29.7%)

Left voluntarily: 6 Changed specialty: 2 Dismissed: 3

More complete follow up not complete Have initiated licensure status and board

certification F/U but too little data to report

Outcomes

Outcome Number

Finished with no concerns 15

Finished with ongoing concerns 10

Voluntarily left 8

Dismissed 3

Resolution not documented 1*

Conclusions?

RIT more common than suspected based on lit. Increased prevalence compared to other studies

? Over-diagnosis For “mild cases”

? Truly higher prevalence Due to population? Due to program characteristics?

All RIT who had concerns in more than 8 areas had ongoing concerns at graduation or did not complete program

RIT who struggled in all 3 performance areas also either did not complete or graduated with ongoing concerns

Professionalism issues very common

Problems Encountered

Information in files varies between programs Missing data

Esp. test scores (fortunately, may be able to acquire) Even within programs, varying data Classification of single most important area for major

problem residents We collected data for an additional group of

residents: those with ITE performance less than 10th %ile, even if no other problems, will follow those as well (n=4 right now)

To come!

Full data reporting (approx 270)

Including USMLE and ITE data

Predictive model?

Follow up data (post-graduation)

Special Thanks to:

Nicole Roberts, PhD data spelunker extraordinaire Reed Williams, PhD Steve Verhulst, PhD Jerry Kruse, MD, MSPH SIU Family Medicine Residency Program Directors:

Janet Albers, MD John Bradley, MD Tom Miller, MD Penny Tippy, MD

Wiley Jenkins, PhD

Referenced Articles

1. Reamy BV, Harman JH. Residents in trouble: an in-depth assessment of the 25-year experience of a single family medicine residency. Fam Med 2006;38(4):252-7.

2. Yao DC, Wright SM. National survey of internal medicine residency program directors regarding problem residents. JAMA 2000;284:1099-1104.

3. Williams RG et al. The nature of general surgery resident performance problems (publication pending)