James S. Powers, MD Josh Denny, MD, MS

Analysis of Pre-Clinical Student Narratives Progress in Assessment of ACGME Core Competencies

Geriatric Competencies

• ACGME Core Competencies (6)• AAMC Geriatric Competencies (26) 8 domains• IM/FP Geriatric Competencies for residents (26/7)• Challenges in defining, operationalizing, and measuring

competencies

Reflection (Narratives)

• Identify educational needs• Develop clinical reasoning• Address professional challenges• Develop skills to cope with clinical complexity

Written Exercises Promote

• Understanding of concepts• Ownership of the learning experience• Enhancement of professionalism and communication

skills• Age appropriate competencies

Narratives-Basis

• Piaget & Erikson's constructive nature of identity• Continuous process of evaluation, selection,

organization “world making”• Communication of feelings and thoughts makes trainee

aware of these• Different type of learning from other coursework

Narratives-Benefits

• Knowledge gained through process of personalization and discernment of values, hopes, fears, actions

• Analysis is qualitative, themes emerge• Provides insight into thinking, maturity, reflective ability• May operationalize competencies

Narrative Example: First Year Medical Student Preceptorship-Small Group Discussion Topics

• Physicians as patients• Ethical dilemmas• Uncertainty• Cultural attitudes and healthcare• Physician lifestyles• Delivering bad news• The student’s role• Patient compliance * First Year VU Medical Students after field experience (N=424)

Innovations

• Knowledge Map – a concept-based curriculum management system

• Learning Portfolio – extends Knowledge Map by to track experience in the clinical years

• POGOe - www.pogoe.org - a free resource of geriatric education products

Natural Language Processing

• Principle: Education documents and most of clinical narratives are in “natural language”

• Goal: We want to harness this text for computer information systemsBetter tracking of what is taughtBetter knowledge of what a trainee saw

• Natural Language Processing (NLP) systems convert these “natural language” human language texts into machine-readable data

9

Document Processing

Document uploaded by lecturer, placed in queue

Document Conversion

Server

KM Concept Identifier

Document Conversion Server pulls next

document off queue, converts to HTML and

Text

Text version placed in

queue

Web serverHTML & PDF versions

Identified concepts indexed for searching

PDF, MS Word, WordPerfect, HTML, PowerPoint, etc

Learning Portfolio

Student types a note in the EMR

Document indexed by

patient

Document indexed by

student

KnowledgeMap Indexer

Database of concepts

Clinical Portfolio

Electronic Medical Record

Appropriate mentors emailed

Section tagging Preprocessor

Mentors evaluate and:1. Give feedback 2. Assign learning objectives

Narrative Example: Geriatric Competency Learning – Medical Clerkship Portfolios

• EMR H&P’s of Third Year Medical Students• E-Learning Portfolio (LP)• E-Knowledge Map catalogue of curriculum (KM)• Detect learning opportunities re: Advance Directives

and Altered Mentation• Messages to student’s Vanderbilt.edu account in real

time with educational prompts

Survey of Residents and Attending physicians

Analysis of student write-ups• Evaluated 142 write-ups using 13-point rating scale• Performance on “Assessment and Plan” ratings improved in electronic

feedback group (p=0.036)

Spickard, III, AS, Gigante J, Stein GS, Denny JC. A Randomized Study of Feedback on Student Write-ups Using an Electronic Portfolio. J Gen Int Med. 2008

Enhanced Narrative Evaluations

• Competency-based “learning opportunities” identified by LP

• Trainee messaged by email with link to KM/POGOe• Reference to: control group, pre-post standardized

MCE, OSCE (SP)

Enhanced Narrative Evaluations

• Operationalize competencies• Evaluate competencies• New EMT tools can be utilized to provide just-in-

time training as well as evaluate trainees at point of patient care

MethodsThree geriatric educators independently reviewed the narratives and graded each core competency topic as present or absent.  These ratings served as the reference to which subsequent computer searches were compared. For each ACGME Core Competency we derived a list search terms that mapped to the vocabulary of the Unified Medical Language System

We searched student narratives, comparing these results with those of the reference ratings of the narratives for each Core Competency

Table 1 A Sample of the UMLS Concepts used for the ACMGE Competency “Systems Based Practice”

Activities of Daily Living Advocating (action)Agencies - Case ManagementAmbulatory care services Arrange for appropriate home care services, as neededAssist patient and/or family in making informed decisions regarding health careCares, Self (Rehabilitation)Case Management ServicesCollaborate with the physician, patient/family/significant others, and other

health team members in planning for continuity of health careDischarge planning Financial costHealth Care Team

ExamplePerhaps the biggest challenge is lack of exposure to these populations. But by having us meet thesepatients and work with them, we may become morecomfortable and competent in caring for them.

ResultsTable 2 KM Competency Searches for MMC Students (n=100 documents,2009) for Second Year Course NarrativesACGME # Docs Experts # Docs KM/LP Sens Spec PPV NPVSBP 27 27 0 Pt Care 12 58 . 56 .45 . 35 .75Profess 73 79 .42 .41 .78 .13PBLI 29 45 .77 0 .94 0Med Know 13 38 .40 .64 .44 .58Comm 15 34 .48 .76 .59 .68

KM/LP KnowledgeMap Learning PortfolioPPV Positive predictive valueNPV Negative predictive value

ConclusionsWe found that UMLS lacks rich vocabulary that clearly describesACGME competencies.  

Embellishment of better search terms will be important as institutions seek authentic and efficient means, such as automated searches of electronic portfolios, to monitor trainees’ progress in attaining ACGME competencies