Competency-based assessment:

The good, the bad, and the puzzling

Kevin W. Eva

Medical Education Assessment Advisory Committee

The Medical Education Assessment Advisory Committee

• Georges Bordage• Craig Campbell• Robert Galbraith• Shiphra Ginsburg• Eric Holmboe• Glenn Regehr

Our charge

1. Defining the “complete” health care professional

“We demand ever more of the latter even as we bemoan its dominance, and yearn for the former while remaining wary of its inefficiencies and lack of uniformity.”

Humanistic expectations vsKnowledge demands

(Anderson, 2011)

19571971

19781980

19841987

19891991

19931995

19971999

20012003

20052007

20092011

0

40

80

120

160

PubMed Search on 'Professionalism, Medical Education'

Year of Publication

# of

arti

cles

Publication of CanMEDs 2000

2. Creating clear and explicit milestones

% prior to blueprint

publication

% after blueprint

publication

p-value

Exam performance 72.5 76.9 nsAgreement that exam tested material taught

63.5 81.3 < 0.01

Agreement that evaluation methods reflected subject matter

60.7 81.5 < 0.01

Agreement that exam was fair

58.0 76.9 < 0.05

(McLaughlin et al., 2005)

General Theme

• Broad, competency-based assessment frameworks, can promote performance improvement rather than simply measuring performance

• They create professional culture through conversation, understanding, and steering

1. Sending the wrong message

Implicit Messages: • Expertise is something that can be

achieved• The goal is to “become independent”

Risk: • Assessments as hurdles• Hesitation to disclose difficulties• Reduced compulsion to offer support

(see MEAAC Report)

Goal TheoryPerformance Orientation Mastery Orientation

•Desire to perform well •Desire to become proficient

•Satisfaction derived from grades

•Deeper engagement

•Greater anxiety •Greater perseverance

•Task avoidance •Stronger motivation

(see Teunissen and Bok, 2013)

2. Ignoring the

continuum

Violation of 3 fundamental laws

VariabilityBig

LosersBig

Winners

Everyone Else

Simple Probability Bell Curve

Society’s Problem

Violation of 3 fundamental laws

VariabilityContext specificity

(Norcini, circa 2006)

“The one truth in medical education.”

Violation of 3 fundamental laws

VariabilityContext specificity

Decay

(Custers and ten Cate, 2011)

General Theme

• Be wary of the unintended consequences of adopting a competence-based assessment framework

(see MEAAC Report)

The Puzzling

To assess competency effectively requires assessment strategies that are …

– broadly focused, – longitudinal, – integrated, – continuous, and – authentic

Three (Overlapping) Themes

• Overcoming unintended consequences• Turning quality assurance into quality

improvement• Ensuring authenticity

1. Overcoming Unintended Consequences

Gist:• Reduce emphasis on exams as point in time

hurdles that prove one’s competence

• Promote notion that trainees are equally accountable for their demonstration of learning

1. Overcoming Unintended Consequences

Strategy:• Build quality improvement activities into assessment

practices

• Use data from licensing process to facilitate formulation of learning plans– And further develop system to enforce follow through

1. Overcoming Unintended Consequences

Examples:• OSCE/CDM components that require candidates to follow-up

on an error made; ask for help; use clinical decision supports

• Feedback intra-candidate relative strengths and weaknesses and require generation of learning plan

• Tailor subsequent assessments to identified weaknesses

2. Turning Quality Assurance into Quality Improvement

Gist:• Reduce the tension between high stakes

licensing assessment and genuine investment in improvement

2. Turning Quality Assurance into Quality Improvement

Strategy:• Further integrate assessment practices across

the continuum of learning with deliberate attention paid to (and reward of) quality improvement

2. Turning Quality Assurance into Quality Improvement

Examples:• Create a formative test tailoring platform for use by

schools/individuals

• Support a national “Diagnostic OSCE” late in UG that can feed data to subsequent stages of training/practice/assessment

• Testing moments that require demonstration of response to data (e.g., OSCE station in which candidates bring personal data)

3. Ensuring Authenticity

Gist:• Assessment that models the realities of

actual practice increases credibility, engagement, and ensures that efforts towards gamesmanship are pedagogically valuable

3. Ensuring Authenticity

Strategy:• Portfolio-supported workplace-based

assessment

• Increasing use of real world supports and real world uncertainties in current practices

3. Ensuring Authenticity

Examples:• Sequential OSCE stations; SPs who are trained to offer

contradictory information mid-station

• Post-encounter probes that require reflection on why approach was appropriate and alternative actions were ruled out

• Internet enabled OSCEs

Completing the puzzle

• Broaden the base of assessment• Build coherent and integrated system• Emphasize the primacy of learning• Harness the power of feedback• Share accountability with the individual

Completing the puzzle

kevin.eva@ubc.ca

Thanks