Consistent Application of

Learning Outcomes in

AssessmentAACOM 2016

Dr. Susan Mackintosh

Dr. Colleen Talbot

Dr. Gerald Thrush

Dr. Kathleen Satterfield

Lara McMahon

Objectives

Describe difficulties in applying/utilizing common

definitions for learning outcomes

Discuss findings from COMP/COMP-Northwest/CPM

study in faculty use patterns of learning outcomes in

assessment practices

Discuss implications and potential future considerations

from preliminary study results

Competency-Based Medical

Education

Flexner report revolutionized medical education

Biomedical model

Evolution to competency-based medical education

Institutional Learning Outcomes

Program Learning Outcomes

Course Learning Outcomes

NBME

NBOME

Learning Outcomes

WesternU Institutional Learning Outcomes (ILO)

Critical Thinking

Breadth and Depth of Knowledge in the Discipline

Interpersonal Communication Skills

Collaboration Skills

Ethical and Moral Decision Making Skills

Life Long Learning

Evidence Based Practice

Humanistic Practice

Learning Outcomes

Program Learning Outcomes

Osteopathic Philosophy and Osteopathic

Manipulative Medicine

Medical Knowledge

Patient Care

Interpersonal and Communication Skills

Professionalism

Practice-Based Learning and Improvement

Systems-Based Practice

Learning Outcomes

Board Learning Outcomes/NBME +OPP

OPP

Behavioral Science

Biochemistry

Biostats and Epidemiology

Cardiovascular

Gastrointestinal

General Principles of H&D

Genetics

Gross Anatomy

Hematopoietic and

Lymphoreticular

Histology and Cell Bio

Micro/Immuno

Musculoskeletal and CT

Nervous System/Special Senses

Nutrition

Pathology

Pharmacology

Physiology

Renal/Urinary

Repro/Endocrine

Respiratory

Curriculum Mapping

Based upon faculty tagging, elegant

curriculum mapping system demonstrates

all ILO’s, PLO’s and BLO’s are covered

both in:

Curriculum delivery

Assessment

Today we are going to focus on

assessment/tagging of exam items

ExamSoft® Exam Summary Analysis

Standard Understanding of Learning

Outcome TermsQuestions were asked:

Are faculty members consistently applying the same

definition(s) of the various competencies?

Are faculty then consistently tagging the appropriate

competency when delivering curriculum and when

assessing curriculum?

Team embarked to do a preliminary study to determine

faculty patterns in tagging of competencies for

assessment

Research hypothesis was that faculty members do not

have a consistent and standardized definition of

competencies

Study Design

Fall of 2013, two healthcare colleges (COMP and CPM)

that share certain curriculum elements began process

where faculty members tagged all exam questions with

appropriate learning outcomes using ExamSoft® CBT

software

Question arose regarding faculty practices in using

standard definitions of competencies in tagging of

questions

COMP and CPM undertook a preliminary study to begin

investigating the application of competency

terminology in assessment

IRB approved protocol

Study Design

Exam questions from the spectrum of pre-clerkship courses were reviewed and 47 questions were selected

All questions had been tagged by original author with ILO’s, PLO’s and BLO’s

Questions embedded in a survey sent to all faculty and faculty were asked to tag each question with any LOs they believed pertinent to the question

32 faculty completed the study and results were compiled in various groups

Clinician vs Basic Medical Science Faculty

College agreement with original author

Consistency amongst the faculty

Results:

Representative Tagging Data for Selected

Questions

Good agreement Poor agreement

Results:

Clinicians vs Basic Scientists

ILO PLO BLO

% M

atc

he

d O

utc

om

es

Results:

COMP vs Podiatric Medicine Faculty

ILO PLO BLO

% M

atc

he

d O

utc

om

es

Results:

Tagging Data for Questions with Multiple LOs

Author: Multiple LO’s Author: Single LO

Results:

Tagging Data for Questions with Multiple LOs

0

20

40

60

80

100

ILO PLO BLO

any LO 2 LO's

ILO PLO BLO

% M

atc

he

d O

utc

om

es

n = 5 n = 4 n = 10

A total of 13 (of 47) questions had multiple LO’s tagged by the author

Q3: Which one of the following molecules is

NOT a direct product of the TCA cycle?

A. ATP

B. CO2

C. FADH2

D. GTP

ILO PLO BLO

Author 2 2 3

Faculty

% Match97% 97% 97%

Faculty

% with Extra LO’s19% 6% 6%

Good agreement in all three LO’s between item author and faculty.

Suggests appropriately tagged item

Q5: Absorption of drug X via the oral route is known to

be saturated (zero order) at doses higher than 100 mg.

If a patient swallowed 300 mg of drug X and assuming

no expenditure of energy is involved in transporting the

drug molecules across the membrane, what would be

the most likely mode of transport for drug X across the

gut?

A. Facilitated transport

B. Active transport

C. Ion-pair transport

D. Simple diffusion

E. Filtration

ILO PLO BLO

Author 2 2 17

Faculty

% Match100% 100% 91%

Faculty

% with Extra LO’s*

53%

(1)9%

62%

(18)

Two of the LO’s had a large number of extra tags Should item have been double tagged?

ILO PLO BLO

Author 1 2 18

Faculty

% Match16% 97% 44%

Faculty

% with Extra LO’s*100%

(2)3%

87%

(14)ILO likely mis-tagged:

100% of the faculty selected ILO 2 and only 16% selected the same ILO as the author

Multiple BLOs likely appropriate:

100% of the faculty selected ILO 2 and only 16% selected ILO 1

Q10: What is the principal function of the cochlea?A. To break complex sound waves into component tones and to transduce these

tones into action potentials in primary sensory fibers.

B. To provide impedance matching by amplifying incoming sound waves as they

move from the oval window to the round window.

C. To filter sound waves so that frequencies below 1000 Hertz are represented at the

base and above 1000 Hertz are represented at the apex.

D. To localize the sound in space by

comparing arrival time in the two ears.

E. To convert efferent signals from the

superior olive into contractions in the

outer hair cells thus amplifying

incoming sound waves.

Q26: A 67-year-old Caucasian woman presents to your office with

2 month history of dry non-productive cough and 10-lbs of

unintentional weight loss. She denies fevers, chills, dyspnea, sore

throat or hemoptysis. Patient has 80- pack year history of smoking.

Family hx is unremarkable. Patients vital signs and physical exams

are unremarkable. A CXR is ordered and shows a peripherally

located solitary pulmonary nodule. Biospy of this solitary mucin

secreting nodule is as shown in image. Which of the following is

MOST LIKELY diagnosis?

A. Adenocarcinoma of the lung

B. Small cell carcinoma of the lung

C. Pulmonary Hamartoma

D. Bronchial Carcinoid

E. Mesothelioma

ILO PLO BLO

Author 2 2 21

Faculty% Match

100% 100% 81%

Faculty

% with Extra LO’s*

78%

(1)47%

(3)81%

(16)

Unanswered Questions/Confounders

Faculty training:

When is it appropriate to tag multiple LO’s

Encourage it more with BLO’s – i.e., many tagged

systems (e.g., renal) should be counter tagged with

an appropriate discipline (e.g., physiology)

How many LO’s is it appropriate to tag?

“serial taggers”

How to minimize mis-tags of LO’s

Unanswered Questions/Confounders

Faculty training:

When is a question testing the ILO’s:

Critical Thinking

Life Long Learning

Evidence Based Practice

Or the PLO’s:

Patient Care

Practice-Based Learning and Improvement

Systems-Based Practice

Next Steps/Future Considerations

Faculty training

Defining the LOs

Understanding which LOs are appropriate for MCQs

Life-long learning

Patient-centered care

Do we need to reassess our tagging?

Do we need to move towards milestones/EPAs?

Are LOs still relevant for the future of healthcare

education?

???

Questions/Discussion

Dr. Susan Mackintosh

smackintosh@western.edu

Dr. Colleen Talbot

ctalbot@western.edu