The Challenges in Surgical Education:The Challenges in Surgical Education:Autonomy, Assessment, & AccountabilityAutonomy, Assessment, & Accountability

Edward D. Verrier, MDEdward D. Verrier, MD

Surgical Director of EducationSurgical Director of Education

Joint Council on Thoracic Surgery EducationJoint Council on Thoracic Surgery EducationJoint Council on Thoracic Surgery EducationJoint Council on Thoracic Surgery Education

K. Alvin K. Alvin MerendinoMerendino Professor of Cardiovascular SurgeryProfessor of Cardiovascular Surgery

University of WashingtonUniversity of Washington

Historical PerspectiveHistorical Perspective

Surgical EducationSurgical Education

•• To vision the future one must acknowledge the To vision the future one must acknowledge the

pastpast

SURGICAL EDUCATION

United States

•• Flexner report 1910Flexner report 1910

–– Carnegie Foundation CommissionCarnegie Foundation Commission

–– “Medical Education in US and Canada”“Medical Education in US and Canada”

–– Embraced Johns Hopkins ModelEmbraced Johns Hopkins Model

•• Recommendations:Recommendations:

Medical School Education Structure

•• Recommendations:Recommendations:

–– Prerequisite EducationPrerequisite Education

•• High school, University (>2 years)High school, University (>2 years)

–– Medical School Requisite Education Medical School Requisite Education

•• 4 years4 years

•• 2 basic science, 2 clinical clerkship2 basic science, 2 clinical clerkship

Post graduate resident training essentialPost graduate resident training essential

•• Apprenticeship model gone except in SurgeryApprenticeship model gone except in Surgery

Remained essentially unchanged for last century

•• William William Halsted,MDHalsted,MD (1903)(1903)–– Formalization of apprenticeship modelFormalization of apprenticeship model

–– Science foundationScience foundation

–– Graduated responsibilityGraduated responsibility

–– Defined structureDefined structure

–– Standardization of trainingStandardization of training

–– Pyramidal system of surgical educationPyramidal system of surgical education

•• William Osler, MD (1908)William Osler, MD (1908)

Early Surgical Education Structure

•• William Osler, MD (1908)William Osler, MD (1908)–– Residents at different levels working togetherResidents at different levels working together

–– Supervision / mentorship by competent facultySupervision / mentorship by competent faculty

–– Bedside teachingBedside teaching

•• Edward Churchill, MD (1933Edward Churchill, MD (1933))–– Eliminate pyramidal systemEliminate pyramidal system

–– Recommended rectangular system of surgical educationRecommended rectangular system of surgical education

–– Mandated ultimately by ACGMEMandated ultimately by ACGME

Robert Hinckley 1853-1945

MGH Ether Dome

First ether anesthesia

Remained essentially unchanged for last century

We believe that in the future, We believe that in the future,

expertise rather than experience, expertise rather than experience,

will underlie competency will underlie competency -- based based will underlie competency will underlie competency -- based based

practice and....certificationpractice and....certification

Aggarwal & Darzi, 2066)

Competency Based Medical Education Competency Based Medical Education

(CBME)(CBME)

•• ....competency based education is an approach to ....competency based education is an approach to

preparing physicians for practice that is preparing physicians for practice that is

fundamentally orientated to graduate outcome fundamentally orientated to graduate outcome

abilities and organized around competencies abilities and organized around competencies abilities and organized around competencies abilities and organized around competencies

derived from an analysis of societal and patient derived from an analysis of societal and patient

needs. It deemphasizes time based training and needs. It deemphasizes time based training and

promises greater accountability, flexibility, and promises greater accountability, flexibility, and

learnerlearner-- centeredness.centeredness.

International CBME Collaborators, 2011

•• Competencies: “Specific knowledge, skills, behaviors and Competencies: “Specific knowledge, skills, behaviors and

attitudes and the appropriate educational experiences attitudes and the appropriate educational experiences

required of residents to complete GME programs.”required of residents to complete GME programs.”

•• GME Competencies:GME Competencies:

–– Patient CarePatient Care

–– Medical KnowledgeMedical Knowledge

What then is competency ?What then is competency ?

ACGME: ~1990 “Outcomes Project”ACGME: ~1990 “Outcomes Project”

–– Medical KnowledgeMedical Knowledge

–– ProfessionalismProfessionalism

–– Interpersonal and Communication SkillsInterpersonal and Communication Skills

–– Practice Practice -- based Learning and Improvementbased Learning and Improvement

–– Systems Systems -- based Practicebased Practice

•• Evolving concepts about competencyEvolving concepts about competency

–– Technical skill ?Technical skill ?

–– Life Long (Maintenance of Certification)Life Long (Maintenance of Certification)

–– Competency Competency vsvs ExpertiseExpertise

–– Milestone Project 2012Milestone Project 2012

Competence - a simple model

Dreyfus Model of Skill Acquisition

The Challenges in Surgical EducationThe Challenges in Surgical Education

AutonomyAutonomy

•• Trainee Trainee -- Trainer DisconnectTrainer Disconnect

–– Trainee:Trainee:

•• Exposure / Experience = CompetencyExposure / Experience = Competency

•• Facts / Data = KnowledgeFacts / Data = Knowledge

–– Google GenerationGoogle Generation

–– Information OverloadInformation Overload–– Information OverloadInformation Overload

•• Adult learnersAdult learners

•• Generational / Gender IssuesGenerational / Gender Issues

•• Work Hour PrioritiesWork Hour Priorities

–– Trainer:Trainer:

The Generational DivideThe Generational Divide

TraditionalistsTraditionalists Baby Baby

BoomersBoomers

Generation Generation

XX

MillennialsMillennials

BirthBirth

YearsYears

Before 1945Before 1945 19461946--19641964 19651965--19801980 19791979--20062006

Business Business QualityQuality Long hoursLong hours ProductivityProductivity ContributionContributionBusiness Business

FocusFocus

QualityQuality Long hoursLong hours ProductivityProductivity ContributionContribution

MotivatorMotivator SecuritySecurity MoneyMoney Time offTime off Time offTime off

Company Company

LoyaltyLoyalty

HighestHighest HighHigh LowLow LowLow

Money isMoney is LivelihoodLivelihood Status Status

symbolsymbol

Means to an Means to an

endend

Today’s Today’s

payoffpayoff

ValueValue Family/CommunityFamily/Community SuccessSuccess TimeTime IndividualityIndividuality

Gender Related Gender Related IIssuesssues

•• “Men are from MarsJ. “Men are from MarsJ.

Women are from Venus”Women are from Venus”11

–– Different reward perspectivesDifferent reward perspectives

–– Different responses to stressDifferent responses to stress–– Different responses to stressDifferent responses to stress

–– Biological / hormonal differencesBiological / hormonal differences

1John Grey, PhD 1992

ANDROLOGYANDROLOGY

The Study of Adult LearningThe Study of Adult Learning

Critical AssumptionsCritical Assumptions

•• The “adult” learnerJJThe “adult” learnerJJ

–– is self is self –– directingJrelatedness to othersdirectingJrelatedness to others

–– enters an educational situation with a great deal of enters an educational situation with a great deal of

experienceexperienceexperienceexperience

–– learn when they perceive a need to knowlearn when they perceive a need to know

–– mmotivated to learn after they experience a need in otivated to learn after they experience a need in

their life situationJproblem focus, task centeredtheir life situationJproblem focus, task centered

–– motivated to learn because of internal factors, not motivated to learn because of internal factors, not

external pressureexternal pressure

•• PROCESS > CONTENTPROCESS > CONTENT

“The Adult Learner;” Knowles et al

Experiential Adult Learning CycleExperiential Adult Learning Cycle

Learning StylesLearning Styles

•• CognitiveCognitive

•• AffectiveAffective

•• PhysiologicalPhysiological•• PhysiologicalPhysiological

•• Interpersonal Interpersonal

The Challenges in Surgical EducationThe Challenges in Surgical Education

AutonomyAutonomy

•• Trainee Trainee -- Trainer DisconnectTrainer Disconnect

–– Trainer:Trainer:

•• No Educational TrainingNo Educational Training

•• Resistant to “Change” by definitionResistant to “Change” by definition

•• Priority issuesPriority issues

•• May face Promotion IssuesMay face Promotion Issues•• May face Promotion IssuesMay face Promotion Issues

•• Many with Huge Many with Huge CClinical loadslinical loads

•• Large administrative LoadLarge administrative Load

–– Trainee:Trainee:

•• Dewey’s “Keys Concepts” of Adult TeachingDewey’s “Keys Concepts” of Adult Teaching

–– ExperienceExperience

•• Process not resultProcess not result

–– DemocracyDemocracy

•• No room for autocracy / harshnessNo room for autocracy / harshness

–– ContinuityContinuity

Critical Concepts in Adult TeachingJ..a Science

–– ContinuityContinuity

•• Logical growth Logical growth

–– InteractionInteraction

•• Bandura’s Extension of Dewey’s Concepts Bandura’s Extension of Dewey’s Concepts

–– Teaching through Inquiry / DiscoveryTeaching through Inquiry / Discovery

•• Self Self –– directed learningdirected learning

•• ProblemProblem--solving learningsolving learning

–– Teaching Through ModelingTeaching Through Modeling

•• Imitation / Identification / ModelingImitation / Identification / Modeling

•• Expertise: Expertise: Power of knowledge and preparationPower of knowledge and preparation

–– Knows something beneficial, knows it well, is prepared to conveyKnows something beneficial, knows it well, is prepared to convey

•• Empathy: Empathy: Power of Understanding and Power of Understanding and Consideration Consideration

–– Realistic understanding of learner needsRealistic understanding of learner needs

–– Adapts to learner’s level of experience and skill developmentAdapts to learner’s level of experience and skill development

Characteristics and Skills of Motivating

Instructors

–– Adapts to learner’s level of experience and skill developmentAdapts to learner’s level of experience and skill development

•• Enthusiasm: Enthusiasm: Power of Commitment and AnimationPower of Commitment and Animation

–– Cares about and values what is being taughtCares about and values what is being taught

–– Appropriate degrees of emotion, and energyAppropriate degrees of emotion, and energy

•• Clarity: Clarity: Power of Language and OrganizationPower of Language and Organization

–– Can be understood and followed by learnersCan be understood and followed by learners

–– Has ability to adapt to second presentationHas ability to adapt to second presentation

•• Timely repetitionTimely repetition

•• Formative feedbackFormative feedback

–– personalpersonal

•• ChoiceChoice

Adult Adult EEducation ducation

Learning BackboneLearning Backbone

–– contentcontent

–– timetime

–– levellevel

–– depthdepth

•• MentorshipMentorship

•• LLearning objectivesearning objectives

The Challenge of Surgical EducationThe Challenge of Surgical Education

AssessmentAssessment

•• Challenge in all medical specialtiesChallenge in all medical specialties

•• Relationship to “Competency”Relationship to “Competency”

•• Part of all Modern Part of all Modern EEducational ducational CCurricular designurricular design•• Part of all Modern Part of all Modern EEducational ducational CCurricular designurricular design

–– Learning Objectives, Curriculum design, Assessment tools, Learning Objectives, Curriculum design, Assessment tools,

Validation, Adoption to certifyValidation, Adoption to certify

•• Learning Learning MManagement Systems (LMS)anagement Systems (LMS)

Purpose of Assessment?Purpose of Assessment?

•• To aid learning through constructive feedback:To aid learning through constructive feedback:

–– Assessment Assessment forfor Learning (formative)Learning (formative)

–– Must be done frequently e.g. WBAMust be done frequently e.g. WBA–– Must be done frequently e.g. WBAMust be done frequently e.g. WBA

•• To check knowledge or skill has been learned:To check knowledge or skill has been learned:

–– Assessment Assessment ofof Learning (summative)Learning (summative)

–– Done infrequently e.g. ExamsDone infrequently e.g. Exams (MSF?)(MSF?)

Classic Surgical Training ModelClassic Surgical Training Model

•• Classic Apprenticeship & ExaminationClassic Apprenticeship & Examination

–– Time basedTime based

–– Minimal training of trainersMinimal training of trainers

–– Formative evaluations: Formative evaluations:

•• In training examsIn training exams

•• Faculty or Program Director evaluationsFaculty or Program Director evaluations•• Faculty or Program Director evaluationsFaculty or Program Director evaluations

•• Little structureLittle structure

–– Summative assessmentSummative assessment

•• Case LogsCase Logs

•• Program Director recommendation of competencyProgram Director recommendation of competency

•• Qualifying examination (cognitive)Qualifying examination (cognitive)

•• Certifying examinations(affective)Certifying examinations(affective)

•• No technical examNo technical exam

•• Do our current methods of assessment guarantee Do our current methods of assessment guarantee

competency?competency?

–– Case NumbersCase Numbers–– High variabilityHigh variability

–– Loose definitions / Little auditLoose definitions / Little audit

–– No established benchmarkNo established benchmark

–– ObservationObservation

Skill AssessmentSkill Assessment

–– ObservationObservation–– InterInter-- and intraand intra--rater variabilityrater variability

–– Case variabilityCase variability

–– No uniform instrumentNo uniform instrument

–– No established benchmarkNo established benchmark

–– Now splitting casesNow splitting cases

–– Summative exams (Qualifying / Certifying)Summative exams (Qualifying / Certifying)–– Little correlation with later success as surgeonLittle correlation with later success as surgeon

–– Remains our benchmarkRemains our benchmark

Accuracy of medical staff assessment of trainee’s

operative performance. Paisley AM et al, Med Teach

2005.

•• What has been tried?What has been tried?•• Simulation testingSimulation testing

–– OSATSOSATS

–– Virtual Simulators (MISTVirtual Simulators (MIST--VR)VR)

•• Structured observationStructured observation

Skill AssessmentSkill Assessment

•• Structured observationStructured observation

–– Live observationLive observation

–– Recorded observationRecorded observation

•• Error detectionError detection

•• Motion trackingMotion tracking

•• Procedure timeProcedure time

Comparing the psychometric properties of

Checklists and global rating scales for assessing

Performance on an OSCE-format examination.

Regehr G. Acad Med 1998

Workplace based

assessment in surgical

training : the UK experience

(so far)

Chris Munsch

Cardiac Surgeon, Leeds UK

Chair, Joint Committee on Surgical Training (2007-2011)

The purpose of assessment� Formative – assessment for learning

� Summative – assessment of learning

Assessing competencies in the new curriculum:

traditional assessment tools

CanMeds MCQs Clinicals Vivas

Medical expert

Communicator

Collaborator

Manager

Health Advocate

Scholar

Professional

Workplace based assessment

tools� Case Based Discussion (CBD)

� Mini clinical evaluation exercise (miniCEX)

� Multisource Feedback (MSF)

� Direct observation of procedural skills (DOPS)

� Procedure based assessment (PBA)

� Linked to Educational Supervisors Report

� Feeds into Annual Review of Competence Progression

PBA Form

CanMeds Curriculum

domains

CBD Mini

CEX

MSF DOPS PBA AES

report

Exams ARCP

Medical expert Knowledge

Clinical Skills

Technical Skills

Communicator Clinical Skills

(professional and

generic)

Collaborator Technical Skills

(professional and

generic)

Manager Knowledge

(professional and

generic)

Health advocate Knowledge

(professional and

generic)

Scholar Knowledge

Professional Clinical Skills

Technical Skills

(professional and

generic)

Faculty Development� Training and Assessment in Practice (TAIP) course

� Webcasts on ISCP website

� Regional delivery of training sessions

� Leader as educator programme

Effective Assessment in CBMEEffective Assessment in CBME

•• Continuous and FrequentContinuous and Frequent

–– Robust onRobust on--going formative feedbackgoing formative feedback

•• Criterion basedCriterion based

–– Milestones or benchmarksMilestones or benchmarks

•• Emphasis on what Trainee will ultimately doEmphasis on what Trainee will ultimately do

–– Robust work Robust work -- based assessmentsbased assessments

•• Quality assessment toolsQuality assessment tools

•• Qualitative measures and methodsQualitative measures and methods

–– Judging portfoliosJudging portfolios

•• Emphasize group wisdomEmphasize group wisdom

–– Activate trainee involvementActivate trainee involvement

The Challenge of Surgical EducationThe Challenge of Surgical Education

AccountabilityAccountability

•• Who will hold us accountable for outcomes, Who will hold us accountable for outcomes,

performance, quality and safety?performance, quality and safety?

–– GovernmentGovernment

–– PayorsPayors–– PayorsPayors

•• Pay for PerformancePay for Performance

–– Medical societiesMedical societies

–– ACGME / Specialty BoardsACGME / Specialty Boards

•• Milestone ProjectMilestone Project

–– Society in generalSociety in general

•• Airline standardsAirline standards

–– OurselvesOurselves

Are We Achieving “Competency” Are We Achieving “Competency”

in Surgical Training in 2011?in Surgical Training in 2011?

Competency

in Surgical Training in 2011?in Surgical Training in 2011?

60

65

70

75

80

85

90

95

100Qualifying examination

Certifying examination

ABS Examination ResultsABS Examination Results

2010

Qualifying exam pass rate:75%

Certifying exam pass rate: 75%

50

55

60

2006 2007 2008 2009 2010

2010

Qualifying exam pass rate: 73%

ABTS Examination ResultsABTS Examination Results

Qualifying exam pass rate: 73%

Certifying exam pass rate: 68%

ExpertiseExpertise

Should we be Striving for Should we be Striving for Should we be Striving for Should we be Striving for

Expertise in Surgical Training in Expertise in Surgical Training in

2012?2012?

•• Must be put into context of evolution of Must be put into context of evolution of

medical educationmedical education

–– Emphasis on competencyJ.expertiseEmphasis on competencyJ.expertise

•• Must meet acceptable outcomes and safety Must meet acceptable outcomes and safety

standardsstandards

Competency

standardsstandards

–– Internally or externally definedInternally or externally defined

•• Must meet societal standardsMust meet societal standards

–– Airline standards Airline standards

Traditional

• Halstedian - “See one, do one, teach one”

• Determined by patient flow

• “Conventional” fixed didactic lectures

• Subjective personal evaluation

• Specific time and place

Surgical Education - Paradigm Shift

• Specific time and place

Next Generation

• Simulation – “Do many, mentored always”

• Each student every variation at own pace

• Interactive, updated (web based) lectures

• Standardized, objective, criterion based evaluation• Formative and Summative

• Continuous at point of clinical care

Thank youThank you

•• Remains time basedRemains time based

•• Assessment / Competency:Assessment / Competency:–– Case logsCase logs

–– Summative examsSummative exams

–– Program Director recommendationsProgram Director recommendations

The Evolving Structure of Resident Education

–– Program Director recommendationsProgram Director recommendations

•• “Integrated” training algorithms increasing“Integrated” training algorithms increasing

–– Plastics / Vascular / CardiothoracicPlastics / Vascular / Cardiothoracic

•• Duty hours limitations real impact in surgeryDuty hours limitations real impact in surgery

•• Discussions of “Criterion” based trainingDiscussions of “Criterion” based training–– Formative evaluationsFormative evaluations

Can We Measure Competency? Can We Measure Competency?

Do We Measure Do We Measure CCompetency?ompetency?

How Should We Measure Competency?How Should We Measure Competency?

•• Time spent in trainingTime spent in training–– Apprenticeship holdover Apprenticeship holdover

•• Log book Log book –– Case numbers, ABTS determinedCase numbers, ABTS determined–– Case numbers, ABTS determinedCase numbers, ABTS determined

•• Program Director recommendationProgram Director recommendation–– Can they say no?Can they say no?

•• Summative examsSummative exams–– Qualifying / Certifying examsQualifying / Certifying exams

•• Formative feedbackFormative feedback–– VariableVariable

Goals and Purposes for Learning

Individual and Situational Differences

Core Adult Learning Principles

1: Learner’s Need to Know

Why, What, How

2: Self Concept of the Learner

Autonomous

Self - Directing

3: Prior Experience of the Learner

Socie

tal g

row

th

Situ

atio

nal D

iffere

nces

Institu

tional gro

wth

Subje

ct M

att

er D

iffe

rences

Adult Learning

3: Prior Experience of the Learner

Resource

Mental Models

4: Readiness to LearnLife related

Developmental task

5: Orientation to Learn

Problem centered

Contextual

6: Motivation to Learn

Intrinsic Value

Personal payoff

Individual Learner differences

Individual growth

Socie

tal g

row

th

Situ

atio

nal D

iffere

nces

Institu

tional gro

wth

Subje

ct M

att

er D

iffe

rences

Knowles MS et al: The Adult Learner; Elsevier, 2005

There are several purposes to formative assessment:There are several purposes to formative assessment:

to provide feedback for teachers to modify subsequent learning activities to provide feedback for teachers to modify subsequent learning activities

and experiencesand experiences

to identify and remediate group or individual deficiencies;to identify and remediate group or individual deficiencies;

to move focus away from achieving grades and onto learning processes, to move focus away from achieving grades and onto learning processes,

in order to increase self efficacy and reduce the negative impact of in order to increase self efficacy and reduce the negative impact of

extrinsic motivationextrinsic motivationextrinsic motivationextrinsic motivation

frequent, ongoing assessment allows both for finefrequent, ongoing assessment allows both for fine--tuning of instruction tuning of instruction

and student focus on progress.“and student focus on progress.“

Feedback is the central function of formative assessment. It typically Feedback is the central function of formative assessment. It typically

involves a focus on the detailed content of what is being learnt rather involves a focus on the detailed content of what is being learnt rather

than simply a test score or other measurement of how far a student is than simply a test score or other measurement of how far a student is

falling short of the expectedfalling short of the expected