Department of Graduate Medical Education (GME) Improving Academic Program Quality by Standardizing...

Department of Graduate Medical Education (GME)

Improving Academic Program Quality by Standardizing and Streamlining the

Evaluation Process for Programs and Faculty

Graduate Medical Education

Stanford University Medical Center Ann M. Dohn, MA

Designated Institutional Official

Nancy Piro, PhD

Education Specialist/ Program Manager


Session Objectives

• The Situation and Problems/Issues: Why Did we Do This?

• What Did We Do?• How Did We Do It?• Results


Why Did We Do This?

• The Situation: – The largest number of citations for our

programs were in the area of evaluations – Program Directors’ Needs Assessment Results

• One of the Top Five Requests for the Department of GME = Templates for Evaluation

– GME Review of Evaluations Results


What Did We Find?

• Poor quality questions• Inappropriate response scales• Questions not core competency based• Evaluations not even being conducted• Program Director frustration with the whole

process


Setting the Stage

• PROGRAM EVALUATION– 82 Programs– 14 Citations

What Can We Do???


What Did We Do? – STEP 1

• Step 1. Standardized Questions– Collaboratively developed Prototype Questions

for Programs to Use for Evaluation of Faculty and Residents• Core Competency Based• Unbiased

– Educated Faculty and Program Directors on Removing Unintended Cognitive Bias from their Evaluations and Evaluating


Tools Were Not Sufficient

• Used group consensus to finalize the questions• Distributed the questions to the Program Directors• Educated them on removing bias

but…. the frustration still remained


What Could We Do? – STEP 2

Step 2. Centralized the Annual Program Evaluation Process.

GME now administers program evaluations for all 82 programs

• Used the Standardized Program Evaluation for :– Faculty– Residents/Fellows


How Is This Implemented?

• GME electronically delivers the evaluations to all their faculty and residents/fellows…– Programs can choose one of two dates –

• Feb or May


Standardized Program Evaluation by Faculty Form


Standardized Program Evaluation by Residents/Fellows Form


Generate reports for each program: 1) Program Eval by Faculty


Generate reports for each program: 2) Program Evaluation by the Residents/Fellows


But still not enough

• Program Directors had the data but didn’t use it…...

SIGH…..


The Total GME Annual Program Review Package

Aggregate Data Delivered “to you”

1. Evaluation results posted for all Program Directors on their Residency Management System website

• Program Eval by Faculty• Program Eval by Residents/Fellows• Internal GME House Staff Annual Survey• ACGME Survey (if available)



2. Review Checklist – All the essentials that need to be covered


GME Website Support



3. Template Agenda for the Annual Review Meeting



• Sign-In Sheet /Documentation for web


Standardized Program Evaluation

• Minutes (Meeting Documentation) Template


Standardized Program Evaluation• As soon as completed, we post the aggregated summary reports

for both Faculty and Residents to our online Residency Management System for each Program:


Standardized Program Evaluation

• Action Plan Documentation Template /Example of an action plan from one of our Programs….


Each program is reviewed by GME • Ensures all the requirements of the APR are completed and

documented:

• …


Outcomes

• 100% compliance in 2010!!


Faculty Evaluations

• At the Dean’s request, GME developed standardized questions for residents and fellows to evaluate their faculty


Annual Academic Evaluation of Faculty


Roadblock

• Could not get PD consensus– Some faculty did not want the results going

directly to the Dean’s office without Chair ‘review’…


Where are we?

• Some programs are using it • The Dean to present to Program Directors


Summative Evaluation

• What is summative evaluation?– A summative evaluation is a method of judging the

competence of a trainee at the end of the program (summation). The focus is on the outcome.

“Assessment with the primary purpose of establishing whether or not performance measured at a single defined point in time meets established performance standards, permanently recorded in the form of a grade or score.”



• Why do we do this? – A program director must provide timely

verification (written documentation) of residency education by completing summative performance evaluations:

1) for all graduating residents/fellows

2) for residents who leave the program prior to completion.

Continued…


Summative Evaluations

3) ACGME also requires this for– residents who are transferring into your

program from another program.

“The program director must obtain written or electronic verification of previous educational

experiences and a summative competency-based performance evaluation of the transferring

resident/intern.”


Summative Evaluations

• GME created Templates for the programs for Program to use in completing Summative Evaluations for their trainees graduating or leaving the program / transferring…


Step 2. Continued…

• Provided Faculty/Program Director Education on the tools and processes– Reviewed the requirements of Summative

Evaluation…



• Why do we do this? – A program director must provide timely

verification (written documentation) of residency education by completing summative performance evaluations:

1) for all graduating residents/fellows

2) for residents who leave the program prior to completion.

Continued…


Summary

• We have found that:– Standardizing Forms and Developing Templates

• Saves a lot of time for our Programs/PDs• Improves feedback to the Programs

– Standardizing Program Evaluations in the GME Office• Not only saves time for the programs, but assures GME that

there will be no citations for not having program evaluations

• Our faculty have been engaged and very positive with development sessions from GME on eliminating bias from their evaluation questions and response scales


Questions?

CONTACT INFORMATION ~Ann M. Dohn, MADesignated Institutional [email protected] Piro, PhDEducation Specialist/ Program [email protected]


Faculty Development SessionEVALUATIONS

38

How do we eliminate unintended bias from our evaluation process?

Our focus today:

Department of Graduate Medical Education (GME) 39

What is cognitive bias…

• Cognitive bias is distortion in the way we perceive reality / information.

• Response bias is a type of cognitive bias which can affect the results of an evaluation if evaluators answer questions in the way they think they are designed to be answered, or with a positive or negative bias toward the fellow being evaluated


Where does response bias occur?

• Response bias most occurs most often in the wording of the question. – Response bias is present when a question

contains a leading phrase or words. – Response bias can also occur in rating scales.

• Response bias can be in the raters themselves– Central Tendency– Halo Effect– Similarity Effect


Examples of Question Bias

• Example 1: "I can always talk to my Program Director about residency related problems."

• Problem: Terms such as "always" and "never" will bias the response in the opposite direction.

• Result: Data will be skewed.



• Example 2: “Career planning resources are available to me and my program director supports my professional aspirations."

• Problem: Double-barreled ---resources and aspirations… Respondents may agree with one and not the other. Evaluator cannot make assumptions about which part of the question respondents were rating.

• Result: Data is useless.



• Example 3: "Communication in my program is good."

• Problem: Question is too broad. If score is less than 100% positive, researcher/evaluator still does not know what aspect of communication needs improvement.

• Result: Data is of little or no usefulness.


Rating Scale Bias

Competence and knowledge in general medicine.

Poor Fair Good Very Good Excellent

The data will be artificially skewed in the positive direction with this scale because there are far more (4:1) positive than negative rating options.


Rater/Evaluator Bias

Response bias can be in the evaluators themselves• Central Tendency• Similarity Effect• Halo Effect


Beware the Halo Effect

• The halo effect refers to a cognitive bias whereby the perception of a particular behavior or trait is influenced by the perception of the former traits in a sequence of interpretations.

• Thorndike (1920) was the first to support the halo effect with empirical research. – People seem not to think of other individuals in

mixed terms; instead we seem to see each person as roughly good or roughly bad across all categories of measurement.


The Halo Effect and Expectations

• The halo effect is involved in Kelley's implicit personality theory– the first traits we recognize in other

people influence our interpretation and perception of later ones because of our expectations.


Halo Effect Extends to Products / Marketing

The iPod has had positive effects on perceptions of Apple’s other products…


Could this impact our evaluations here?

Empirical evidence from our HouseStaff…. A question from the most recent GME HouseStaff survey:

“The general feeling in my program is that your ability will be labeled based on your initial performance.”

54.7%

24.4%

15.6%

71.8%

17.9%

10.3%

Overall SHC-LPCH Overall Peds Fellows


Reverse Halo Effect

• A corollary to the halo effect is the reverse halo effect (devil effect) –individuals, brands or other things

judged to have a single undesirable trait are subsequently judged to have many poor traits, allowing a single weak point or negative trait to influence others' perception of the person, brand or other thing in general.


Blind Spots

• In the 1970s, the social psychologist Richard Nisbett demonstrated that we may have no awareness of when the halo effect influences us (Nisbett, R.E. and Wilson, T.D., 1977)

The problem with Blind Spots is that we are blind to them…

Department of Graduate Medical Education (GME) Improving Academic Program Quality by Standardizing...

Documents

Transcript of Department of Graduate Medical Education (GME) Improving Academic Program Quality by Standardizing...