Computer Assisted Coding and Beyond An Academic's Adventures with Clinical Natural Language...

Computer Assisted Coding and Beyond

An Academic's Adventures with Clinical Natural Language Processing in the Real World

Philip Resnik

University of Maryland

CodeRyte, Inc.

Who am I?

Why am I here?

BillingMedical Coding

CPT (procedure codes)ICD (diagnosis codes)

PayersPayers

DenialsResubmissions

ORDER_EXAM: MRI Hd wo&w ORDER_IND: head - h/o plasmacytoma^ MR head, without and with IV gadolinium. Comparison is made with previous outside…

Transcription

Medical coding landscape

Source: Healthcare Revenue Cycle Management, Triple Tree, http://www.triple-tree.com/ResearchRequest.aspx?researchId=13

Medical coding: a few highlights

• O($100B) per year in U.S. healthcare costs

• Painful shortage of qualified medical coders

• Overhaul of coding standards coming in 2013

• Under-coding = lost revenue

• Over-coding = fines or even criminal charges

All this over a mere document annotation problem?

A dream problem for NLP

• Language data– 2 billion doctor-patient encounters per year

• Annotated language data for supervised methods– A by-product of the existing process (cf. MT)

• Rich existing knowledge sources– Lexicons, ontologies

• Humans in the loop– Value in both automation and in human efficiencies

• A real-world task based purely on the text– Use of world knowledge not just avoided, but illegal?!

Getting started

• Learned what coding was about!

• Started with ~300,000 coded radiology notes– HIPAA Privacy Rule not yet in effect

• Investigated existing knowledge resources

• Engine prototype development

The academic’s adventure begins…

• Off the shelf resources? Often not a good match.– Extreme variability in the input (e.g. “wet/dry”)

– New resources needed to be built out

• We have annotations ≠ We understand the task– Coding expertise crucial, both art and science

– Inter-coder agreement an issue (more on this later)

• Context for the NLP is crucial– UI and user experience

– Workflow

• Did I say prototype? Silly me, I meant product!

BillingMedical Coding

PayersPayers


Transcription

Medical coding landscape

Billing

PayersPayers


Transcription

Computer assisted coding landscape

NLP Engine Routing Coder Review

Mrs. Zoe is a 57-year-old female who has been having chest pains which she describes as a sharp pain, located substernally occurring at night when she tries to lie on her right side. She has not had any exertional type chest discomfort and the discomfort in her chest will last as long as she is lying in that position. …

Mrs. Zoe exercises daily, walking one and a half to three miles and also uses some weights. Her mother is age 81 and has a history of angina and congestive heart failure along with atrial fibrillation.

Weight is 150 pounds, stable. No history of thyroid dysfunction. No renal dysfunction. No gastrointestinal symptoms. No asthma, wheezing, or lung problem. Is having menopausal symptoms. No claudication. Neurologic is negative.

Identifying document regions

Mrs. Zoe is a 57-year-old female who has been having chest pains which she describes as a sharp pain, located substernally occurring at night when she tries to lie on her right side. She has not had any exertional type chest discomfort and the discomfort in her chest will last as long as she is lying in that position. …

Mrs. Zoe exercises daily, walking one and a half to three miles and also uses some weights. Her mother is age 81 and has a history of angina and congestive heart failure along with atrial fibrillation.

Weight is 150 pounds, stable. No history of thyroid dysfunction. No renal dysfunction. No gastrointestinal symptoms. No asthma, wheezing, or lung problem. Is having menopausal symptoms. No claudication. Neurologic is negative.

History of present illness

Past medical history

Family history

Review of systems

15

Some other kinds of regions

• Negated– No evidence of pneumonia

• Equivocal or Modal– … could represent atelectasis…

– … likely fracture…

16

Sentence breaking

Mrs. Zoe is a 57-year-old female who has been having chest pains which she describes as a sharp pain, located substernally occurring at night when she tries to lie on her right side. She has not had any exertional type chest discomfort and the discomfort in her chest will last as long as she is lying in that position. …Mrs. Zoe exercises daily, walking one and a half to three miles and also uses some weights. Her mother is age 81 and has a history of angina and congestive heart failure along with atrial fibrillation. Weight is 150 pounds, stable. No history of thyroid dysfunction. No renal dysfunction. No gastrointestinal symptoms. No asthma, wheezing, or lung problem. Is having menopausal symptoms. No claudication. Neurologic is negative.

17

Morphological analysis

Mrs. Zoe is a 57-year-old female who has been having chest pains which she describes as a sharp pain, located substernally occurring at night when she tries to lie on her right side.

= pain + PLURALIn this context, pains is the same as pain.

Sometimes singular vs. plural matters, e.g. cyst is different from cysts.

18

Approaches to identifying/combining information units

19

Creating internal representations of evidence

Symptom: pain

Degree: sharp

Loc: chest

LocMod: substernal

Source: HPI

…

Chest pains which she describes as a sharp pain...

Sharp pain in her chest…

Sharp chest pain…

Chest pain which feels sharp…

20

Mapping to/predicting codes

• Rule-based matching– Match representation assign code

• Statistical prediction– Statistical prediction of code based on aggregated data

Symptom:pain

Degree: sharp

Loc: chest

LocMod: substernal

Source: HPI

…

Coding logic

• General logic– “Most certain” coding guidelines– Pertinent vs. incidental findings– Choice of primary code– Code combination

• Client or payer-specific logic– Resolution of gray areas– Pseudo-codes

22

Routing based on confidence

Coded note

P(correct) > τ

Code queue

Review queue

Confident queue

QAQABilling

Review carefully: likely missing

documentation

Ok to send straight to

billing

Review: Coded but does not meet

confidence thresholds

Billing

NLP Engine Routing

Engine Coding versus Confidence Assessment

Metadata evidence

Language evidence

Looking at all the evidence in the chart, which codes are the best choice?

Looking at chosen codes and how the evidence led to them, how confident are we that those codes are correct?

Sufficiently confident

“Coder” “Auditor”

24

Pr(Correct | Codes, Evidence, Explanation)Pr(Codes| Evidence)

Academic’s adventures continued…The input

What we’re used to What we find

A single XML input file Metadata/data integrations

Everyone is named “Johnson” Initials: SOB; Dr. Parkinson; …

Consistent region headers Region titles vary or are absent

Limited samples (e.g. i2b2: chest x-ray, renal procedures)

Full range of ICDs across radiology, pathology, orthopedics, cardiology, …

Academic’s adventures continued…Defining the task

What we’re used to What we find

Static code inventory Regular updates

Stable coding guidelines Guideline changes

Uniform performance target Targets varying by customer type

Aggregate evaluation Focus on individual errors

Academic’s adventures continued…Market misconceptions

• Typical confidence assessment settings:– CPT accuracy 97-99%

– ICD accuracy 92-95%

• Inter-coder agreement– CPT 90%

– ICD 58%

• Intra-coder agreement– CPT 96%

– ICD 68%

Resnik et al. (2006) Using Intrinsic and Extrinsic Metrics to Evaluate Accuracy and Facilitation in Computer Assisted Coding

The market does not really “get” the notion of human upper bounds.

Low inter-coder agreement is an open secret in the coding community.

Academic’s adventures continued…Market misconceptions

Rule-based methods

Rule-based methods

Statistical NLP

Machine learning

Rule-based methods informed by large scale

data analysis

Data

29

Healthcare (the parts I’ve seen) has not, on the whole, caught up to the state of the art in NLP.

“Coding is about following the rules, so it’s better to use rule-based methods”

“An engine using machine learning isn’t using ‘pure’ NLP”

Mr. John Roe was seen in our office today in follow up of his paroxysmal atrial fibrillation. As you know, he is a 57 year old gentleman who had electrical cardioversion in May 2002 and had been maintained on Betapace since that time. His last visit in our office was July 23, 2003. He recently called our office in February stating he was back in atrial fibrillation which was documented on electrocardiogram. I elected to increase his Betapace to 160 mg twice a day and he did convert back to normal sinus rhythm. We had recommended Coumadin to him at that time but he did not start any Coumadin. He has done well since with no recurrence of arrhythmia and he is acutely aware of when he goes into the fibrillation. He denies any shortness of breath, chest discomfort of congestive heart failure symptoms and has otherwise felt quite well. His only medication is the Lexa pro 10 mg a day as an antidepressant and the Betapace. His review of systems is otherwise unchanged and negative.

Source: http://www.nextgen.com/images/screenshots/card01.jpg

The push for structured data entry

“This system is designed for physicians to point and click their way through an entire exam quickly and effortlessly.” (NextGen EMR product review)

Question 1

• How will structured data entry affect clinical knowledge discovery?

ORDER_EXAM: MRI Hd wo&w ORDER_IND: head - h/o plasmacytoma^ MR head, without and with IV gadolinium. Comparison is made with previous outside MR head examinations 5/3/04 and 11/16/04. On the earliest outside examination, there was a mass in the right central skull base, extending infratemporal fossa, sphenoidsinus, and foramen ovale. This subsequently was demonstrated to represent a plasmacytoma. This mass is markedly reduced in size on the subsequent outside MR. Our examination continues to show abnormal signal and peripheral enhancement, in the right central skull base, and involving right clivus, right sphenoid, and base of right pterygoid. This is probably stable when compared with 11/16/04, but is considerably smaller than 5/3/04. The infratemporal soft tissue component of the lesion has resolved. No new or progressing bone lesion. Incidental note is made of a small amount of hemosiderin deposition within the cortex of the left parietal operculum without abnormal enhancement. This could represent cryptic vascular malformation, or chronic lacunar infarct. Mild cerebral leukoaraiosis. …

Clinical history:

History of plasmacytoma (head)

Mass in right central skull base

Mass subsequently reduced in size

Findings:

Abnormality in right central skull base…

Mass smaller in size

Mild cerebral leukoaraiosis

Current medications:

Melphalan 9 mg/m2 per day

Prednisone 100 mg/day

plasmacytoma

steroidcancer

drug

prednisone

disease

treats

Hypotheses:

Cryptic vascular malformation

Chronic lacunar infarct

Information

• The process of knowledge discovery is a natural cycle

• At every iteration, information emerges from data by structuring and categorizing the data according to what we know now

• As we improve our knowledge, those structures and categories change

Data

KnowledgeKnowledge

Data

Information

37

ORDER_EXAM: MRI Hd wo&w ORDER_IND: head - h/o plasmacytoma^ MR head, without and with IV gadolinium. Comparison is made with previous outside MR head examinations 5/3/04 and 11/16/04. On the earliest outside examination, there was a mass in the right central skull base, extending infratemporal fossa, sphenoidsinus, and foramen ovale. This subsequently was demonstrated to represent a plasmacytoma. This mass is markedly reduced in size on the subsequent outside MR. Our examination continues to show abnormal signal and peripheral enhancement, in the right central skull base, and involving right clivus, right sphenoid, and base of right pterygoid. This is probably stable when compared with 11/16/04, but is considerably smaller than 5/3/04. The infratemporal soft tissue component of the lesion has resolved. No new or progressing bone lesion. Incidental note is made of a small amount of hemosiderin deposition within the cortex of the left parietal operculum without abnormal enhancement. This could represent cryptic vascular malformation, or chronic lacunar infarct. Mild cerebral leukoaraiosis. …

Clinical history:




Findings:







plasmacytoma

steroidcancer

drug

prednisone

disease

treats

Hypotheses:



Information

• The process of knowledge discovery is a natural cycle

• At every iteration, information emerges from data by structuring and categorizing the data according to what we know now

• As we improve our knowledge, those structures and categories change

Data

KnowledgeKnowledge

Data

Information

38

the knowledge discovery cycle is broken.If the full clinical narrative never comes into existence,

39

Clinical history:




Findings:







plasmacytoma

steroidcancer

drug

prednisone

disease

treats

Hypotheses:



Information

Clinical history:




Findings:





QD100 mgprednisone

QD9 mgmelphalan

FRQDOSENAME

Knowledge

Without the original data, we can never reanalyze physicians’ observations in the light of new

knowledge and new categories.

Question 2

• How will structured data entry affect clinical communication between physicians?

I worry that EMRs as implemented can actually downgrade the quality of information passed between health care teams

– Henry F. Smith, Jr., MD, “EMRs: Finding a balance between billing efficiency and patient care", Commentary, The Times Leader, Wilkes-Barre, PA, June 12, 2011.

Previous EHR studies

• Adoption

• Cost

• Economic value

• Quality of care metrics

• No previous study directly compares clinical communication using free dictations with clinical communication using EHRs.

An idealized study



Clinically relevant information in dictation

Clinically relevant information in structured entry

A practical approximation



Materials

• Source material: 2000 cardiology notes

• Sample: 20 notes– Starting at a randomly chosen point

– All different patients

– Mostly different physicians

• Also used a small, separate set of notes for development

Results(a) Both experts (b) MD Cardiologist

0% 50% 85% 55% 85% 100%

0 0.6 1.85 1.45 3.45 5.25

Seriousness of omission

Percentage of documents

Average omissions per document

Considered an omission only if both experts identified it as an omission

• Some omissions seem straightforward to remediate with easy changes to the EHR specification, e.g.– Negative patient reports (“denies SOB”)

– Degrees for symptoms (“mild/severe pain”)

– Reactions to allergies (“rash/hives”)

Results, disregarding “remediable” omissions

(a) Both experts (b) MD Cardiologist

0% 25% 50% 45% 80% 95%

0 0.3 0.95 0.75 2 2.65

Seriousness of omission

Percentage of documents

Average omissions per document

Considered an omission only if both experts identified it as an omission

• Other omissions seem difficult to remediate, even in principle– Nuanced/detailed elaborations

• “almost brought to tears just in getting her up on the examination table”

• “able to walk on flat levels and walk at a moderate pace for one hour without abnormal shortness of breath or chest pain”

• Other omissions seem difficult to remediate, even in principle– Temporal/logical context

• ventricular tachycardia occurred “during post myocardial infarction care…far removed from the time of [patient’s] infarction”

• the dictating physician was “hesitant to recommend [patient’s] FAA certification renewal” without a repeat of a previous catheterization

• Other omissions seem difficult to remediate, even in principle– Dictating physician’s thought process

• recommends continuing Toprol because it “seems to be controlling [the patient’s] palpitations well”

• considers discomfort to be “suggestive of angina”

• believes that results of stress testing “would rule out significant major coronary artery disease, despite it being a somewhat incomplete study”

Difficult to remediate: the things that make the clinical narrative a narrative

“In an EHR the story is lost. In a patient record you have a story and you just follow along in the text of the story and you can see what happened…”

– Coder in “Coding Professionals’ Feelings toward Computers and Automated Coding”, a 2007 qualitative study by Mary Stanfill, vice president for HIM practice resources at AHIMA

…As EMRs proliferate, and increased Medicare scrutiny looms, medical documentation is evolving from its original goal of recording what actually was going on with a patient, and what the provider was actually thinking, to sterile boilerplate documents designed to justify the highest billing codes.

…In years past, a well-written history and physical, or progress note, would unfold like a story, giving a vivid description of the patient’s symptoms and physical exam at the point of the encounter, as well as the synthesis of the data and the plan of care.

– “EMRs: Finding a balance between billing efficiency and patient care", Henry F. Smith, Jr., MD, Commentary, The Times Leader, Wilkes-Barre, PA, June 12, 2011.

If you lose the language, you lose the story.

A dilemma

• There are genuine advantages to structuring the representations in clinical records.

• But, I have argued, structured data entry:– destroys the knowledge discovery cycle– can omit information that clinicians need in order

to communicate effectively, with potentially serious consequences for patient care.

NLP Engine

Clinicians

Researchers

Policy makers

Patients

plasmacytoma

steroidcancer

drug

prednisone

disease

treats

ORDER_EXAM: MRI Hd wo&w ORDER_IND: head - h/o plasmacytoma^ MR head, without and with IV gadolinium. Comparison is made with previous outside MR head examinations 5/3/04 and 11/16/04. On the earliest outside examination, there was a mass in the right central skull base, extending infratemporal fossa, sphenoidsinus, and foramen ovale. This subsequently was demonstrated to represent a plasmacytoma. This mass is markedly reduced in size on the subsequent outside MR. Our examination continues to show abnormal signal and peripheral enhancement, in the right central skull base, and involving right clivus, right sphenoid, and base of right pterygoid. This is probably stable when






Transcription

Clinical history:




Findings:







Physicians focus on the care of the patient and communicate unimpeded, full, narrative clinical data.

Informed by the best current knowledge and data, language technology transforms clinical language into standardized, interoperable, available information.

Both health information technology and medical communities of practice inform, and are informed by, evolving medical knowledge.

A way forward: recognizing thatstructured representations ≠ structured input

Closing thoughts

• Clinical text is potentially a gold mine of data.

• Real-world clinical NLP – Quickly gets beyond the academic comfort zone.

– Can potentially revolutionize healthcare.

• Naturally occurring clinical language is in danger.– This community can help save it.

Thank you!

Thanks.

Much obliged.

Appreciate it.

Thx

Tx

Ta!

Cheers!thank you

(conversational expression of gratitude)

thanks

(acknowledgment of appreciation)

acknowledgment

(statement acknowledging something or someone)

Mr. John Doe was seen in our office today in follow up of his paroxysmal atrial fibrillation. . . . He recently called our office in February stating he was back in atrial fibrillation which was documented on electrocardiogram. I elected to increase his Betapace to 160 mg twice a day and he did convert back to normal sinus rhythm. We had recommended Coumadin to him at that time but he did not start any Coumadin. He has done well since with no recurrence of arrhythmia and he is acutely aware of when he goes into the fibrillation. . . .He seems to be doing well on the increased dose of Betapace 160 mg twice a day. I told him he should take a daily baby aspirin and also that if he has recurrent episodes of fibrillation, he needs to let us know because I think he would need to be on Coumadin anticoagulation and may need an adjustment in his antiarrhythmic regimen.

If the full clinical narrative never comes into existence…

There is clear evidence that this patient’s self-reports are trustworthy and relevant. In your thinking on his clinical care, you

should make sure to pay attention to them.

Here’s the reasoning connected to my recommendation of Coumadin, the status of that recommendation, and the circumstances

under which I think the recommendation should be revisited.

60Philip Resnik, NLP Tutorial

ORDER_EXAM: MRI Hd wo&w ORDER_IND: head - h/o plasmacytoma^ MR head, without and with IV gadolinium. Comparison is made with previous outside MR head examinations 5/3/04 and 11/16/04. On the earliest outside examination, there was a mass in the right central skull base, extending infratemporal fossa, sphenoidsinus, and foramen ovale. This subsequently was demonstrated to represent a plasmacytoma. This mass is markedly reduced in size on the subsequent outside MR. Our examination continues to show abnormal signal and peripheral enhancement, in the right central skull base, and involving right clivus, right sphenoid, and base of right pterygoid. This is probably stable when compared with 11/16/04, but is considerably smaller than 5/3/04. The infratemporal soft tissue component of the lesion has resolved. No new or progressing bone lesion. Incidental note is made of a small amount of hemosiderin deposition within the cortex of the left parietal operculum without abnormal enhancement. This could represent cryptic vascular malformation, or chronic lacunar infarct. Mild cerebral leukoaraiosis.

The role of computer assisted coding

• Consistency– Coders produce more consistent output using CAC than when

coding from scratch (Resnik et al. 2006).

• Reporting, analytics, and flagging problems– Getting ICD-10 codes retrospectively on previously coded ICD-9

data for outcomes analysis, triggering rules, etc. (The world doesn’t start over with ICD-10!)

• Coder and clinician training– Showing ICD-9 and ICD-10 together– Deficiency analysis for dictations

The role of computer assisted coding

• The network effect: using machine learning methods to aggregate coder expertise– Capturing language variability

– Identifying patterns of coder error

– Identifying patterns of machine error• Principled direct-to-bill choices (cf. Jiang et al., 2006)

Computer Assisted Coding and Beyond An Academic's Adventures with Clinical Natural Language...

Documents

Transcript of Computer Assisted Coding and Beyond An Academic's Adventures with Clinical Natural Language...