March 2, 2004: I. Sim Clinical Decision Support Systems Medical Informatics Clinical Decision...

March 2, 2004: I. Sim Clinical Decision Support SystemsMedical Informatics

Clinical Decision Support Systems

Ida Sim, MD, PhD

March 2, 2004

Division of General Internal Medicine, and the Program in Biological and Medical Informatics

UCSF

Copyright Ida Sim, 2004. All federal and state rights reserved for all original material presented in this course through any medium, including lecture or print.


Guest Lecture

• Thursday, March 4, 1:30 to 3pm• Paul Tang, MD Chief Information Officer, Palo

Alto Medical Foundation– state of the art electronic medical record

• same one that Kaiser is spending $2.8 billion on

– the promise and the reality


Outline

• The quality chasm• Clinical decision support systems (CDSS)

– definition– how they work

• Effectiveness of CDSSs– improving quality– reducing errors

• Fundamental barrier


Crossing the Quality Chasm

• Quality care for the 21st century • safe, effective• patient-centered, timely, efficient• equitable

• Evidence-based practice is means to quality– practice based on currently best available

evidence from clinical research


Hardest Way to Practice

• Over 1,000 guidelines in National Guideline Clearinghouse

• Over 4,600 journals indexed in Medline– over 10,000 RCTs per year– over 2700 systematic reviews per year

• Logistically impossible to “keep up”– let alone make sense of and apply the evidence


Informatics to the Rescue• Information technology touted to improve

quality of care– legible information more available

• chart, lab results, allergies

– care is more efficient• visit level coding, charting using macros, e-

prescribing

– improvement in intermediate measures• vaccination and screening rates

– improvement in patient outcomes• ...


Outline






What is a CDSS?

• Software that is designed to be a direct aid to clinical decision-making

• in which the characteristics of an individual patient are matched to a computerized clinical knowledge base

• and patient-specific assessments or recommendations are then presented to the clinician and/or the patient for a decision (Sim et al, JAMIA, 2001)

• Ground principle: CDSSs should be based on best current evidence, which continually evolves


What Isn’t a CDSS

• Medline• UpToDate• Static guideline repositories

– www.guideline.gov (National Guideline Clearinghouse)

• Online laboratory data, test results, chart notes• Retrospective quality improvement reports

– how your vaccination rates compare to your colleagues’


Major Target Tasks of CDSSs• Diagnostic support

– DxPlain, QMR• Drug dosing

– aminoglycoside, theophylline, warfarin• Preventive care

– reminders for vaccinations, mammograms• Disease management

– diabetes, hypertension, AIDS, asthma• Test ordering, drug prescription

– reducing daily CBCs in hospital, drug allergy checking• Utilization

– referral management, clinic followup


How Do CDSSs “Think”?• Rule-based• Bayesian network

– formal probabilistic reasoning, extension of decision analysis

• Neural network• Adhoc (non-mathemetical reasoning about probability)

– e.g., if high WBC AND cough AND fever AND abn. CXR then likelihood of pneumonia is 4 out of 5

• e.g., DxPlain, QMR• Fuzzy logic, genetic algorithms, case-based

reasoning, etc., or hybrids of these


Rule-Based Approaches (1)

• Forward chaining/reasoning (data-driven)– start with data, execute applicable rules, see if new

conclusions trigger other rules, and so on– example

• if HIGH-WBC and COUGH and FEVER and ABN-CXR => PNEUMONIA

• if PNEUMONIA => GIVE-ANTIBIOTICS• if GIVE-ANTIBIOTICS => CHECK-ALLERGIES• if PNEUMONIA and GIVE-ANTIBIOTICS and NOT

(ALLERGIC-DOXYCYCLINE) => GIVE-DOXYCYCLINE

– use if sparse data


Rule-Based Approaches (2)

• Backward chaining/reasoning (goal-driven) – start with “goal rule,” determine whether goal rule is

true by evaluating the truth of each necessary premise– example

• patient with lots of findings and symptoms• is this SLE? => are 4 or more ACR criteria satisfied?

– malar rash?– discoid rash?– skin photosensitivity? etc

• if SLE => ...– use if lots of data


Problems with Rule-Based CDSSs• A CDSS “knowledge base” has many rules• Problems

– need a LOT of rules, one for each contingency• if MOD-WBC and COUGH and FEVER and ABN-CXR =>

PNEUMONIA

– rules may be contradictory• if COUGH and ABN-CXR => INTERSTITIAL-LUNG-DZ

– rules may be circular– maintenance of knowledge base over time, etc.

• Need knowledge engineering (AI) and clinical expertise to build and maintain the KB

• What about sharing KBs?


Medical Logic Modules (MLMs)

• help_amp_for_pneumonia - Ampicillin for Pneumonia

• maintenance:– title: Ampicillin for

Pneumonia;;– filename:

help_amp_for_pneumonia;; – version: 1.00;; – institution: LDS Hospital;; – author: Peter Haug, M.D.;

George Hripcsak, M.D.;; – specialist: ;; – date: 1991-05-28;;

• validation: testing;; • library:

– purpose: Recommend the use of ampicillin for pneumonia.;;

– explanation: If the patient has pneumonia, then suggest treatment with ampicillin unless there is a penicillin allergy.;;

• keywords: pneumonia; penicillin; ampicillin;;

• citations: 1. HELP Frame Manual, version 1.6. LDS

Hospital, August 1989, p.81.;;

• Forward chaining rules for CDSSs• Expressed in Arden Syntax (an international ASTM standard)


Sharing of MLMs: No Success• Work of reuse often greater than building from

scratch– rules are often outdated: need to check evidence base– context is under-specified

• is pneumonia rule inpatient or outpatient? in HIV patients?

– can be wrong for local context• resistance patterns vary in different locales

– definitional problems• your “pneumonia” is not my “pneumonia”

– curly braces problem• if {K+} > 5.5 => alert MD• how to access the value of K+ automatically? requires interfacing

to lab system which differs from place to place


How Do CDSSs “Think”?• Rule-based• Bayesian network

– formal probabilistic reasoning, extension of decision analysis

• Neural network• Adhoc (non-mathemetical reasoning about probability)

– e.g., if high WBC AND cough AND fever AND abn. CXR then likelihood of pneumonia is 4 out of 5

• e.g., DxPlain, QMR• Fuzzy logic, genetic algorithms, case-based

reasoning, etc., or hybrids of these


Neural Networks• Example of a data-driven data mining method• Finds a non-linear relationship between input parameters and

output state• Structure of network

– usually input, output, and 1-2 hidden fully connected layers

– each connection has a “weight”


Neural Network for MI Diagnosis

• Inputs (e.g., all patient characteristics in the EMR) • EKG findings (ST elevation, old Q’s)• rales (Yes, No)• JVD (in cm)

• Outputs are the set of possible outcomes/diagnoses

EKG findings

Rales

JVD

Response to TNG

Acute MI

No Acute MI


Training the Neural Network

• Network gets “trained” – feed network many examples of known patients and their

diagnoses– system iteratively adjusts the weights of the connections

to find the network “pattern” that associates sets of input variables (patients) with the right output state (MI or not)

• Test network’s accuracy on another set of patients• In Baxt’s MI neural network

– training set: 130 pts with MI, 120 without– test set: 1070 UCSD ER patients with anterior chest pain


Baxt’s Acute MI Neural Net• Evaluation results: prevalence of MI 7% (Lancet, 1996)

• Results were driven by non-standard predictors– rales, jugular venous distention

• Why isn’t this neural network used more widely?– “black box” nature limits explanatory ability and lessens

acceptance– users have to input the variables manually

• if EMRs more widely available, these types of systems may be more prevalent

Sensitivity Specificity

Physicians 73.3% (63.3-83.3) 81.1% (78.7 – 83.5)Neural Net 96.0% (91.2 – 100) 96.0% (94.8 – 97.2)


CDSS Methods• Vast majority of clinically-used CDSSs use rule-

based reasoning– problem of combinatorial explosion of rules

• Major limitations– how to represent some data (e.g., “looks sick”)– formal, reproducible methods for making clinical

decisions

• Other major limitation is source of input data– manual input of data by doctors will not work– EMR can enable a new era of CDSSs

• But lots can be done with current technology


Outline




• Fundamental barriers


CDSS Effectiveness

• In controlled trials, only occasional modest benefit found (Hunt, JAMA 1998; updated RB Haynes 2000)

– diagnosis: 1/5 studies beneficial– drug dosing: 9/15– preventive care reminders: 19/26

• Few studies looked at patient outcomes– 6 of 14 showed benefit


Shortcomings of CDSS Eval Literature

• Variable study quality– 35% rate >8 on 10 point quality scale (mean ~6.2)– more recent studies better quality

• Low power– 5 of 8 studies of patient outcome had low power

• Unit of randomization was the patient– physicians treated some patients with CDSS, and some

without CDSS– this would tend to …. any effects of the CDSS

• Probably publication bias


Shortcomings of Study Designs

• E.g., evaluate a CDSS for hypertension treatment• Is RCT best design for determining effectiveness?

– should randomize MDs, usually low power– intervention is usually more than just the CDSS

• e.g., “buy-in” sessions to HTN guideline underlying CDSS

– limited generalizability• applies only to this particular CDSS• integration of CDSS into existing workflow often unique to

study site

– if CDSS shows no effect, standard RCT gives little insight into why


Heterogeneity of CDSSs

• Preventive care reminder CDSS– A clerk routinely abstracts preventive care interventions

from paper chart into a database. Before each clinic session, nurse runs the CDSS for patients coming in that day. Guideline-based recommendations are printed out on paper and attached to front of chart. Doctor orders preventive care in usual way using paper-based methods.

• Hypertension treatment CDSS– Clinic has an EMR. During patient visit, CDSS notes that BP

and trend is too high. It checks patient’s Cr, diabetes status, cardiac status, current meds and allergies and recommends drug therapy change according to JNC VI guidelines. Presents e-prescription for MD to verify. If verified, order is sent directly to pharmacy and medication list updated.


Understanding CDSS Effectiveness• Hunt counted the number of systems in each category

(e.g., drug dosing) that were “effective”• How would you improve on Hunt’s systematic review?

– CDSSs are very heterogeneous– does the heterogeneity explain any variation in benefit?

• How to meaningfully characterize CDSSs?– target decision maker(s) (MD, nurse, patient)– urgency of decision (stat result, outpatient screening)– method of delivery (paper, EMR, pager)– force of recommendation (suggestion, requirement) ...


Typology of CDSSs

CONTEXTTarget task typeTarget patient settingPoint of careWorkflow integration

Data source-system

intermediary

CDSSCustomizationClinical knowledge source Update mechanismUnit of analysis

INPUTData source

OUTPUTAction complexityAction embeddedCompliance urgencyForce of action recommendationRecommendation explicitness

ProcessorTarget decision maker

System user

System-user interface

System user/ Processor/Target decision maker

OR• F

orm

/Mod

e of

info

rmat

ion

gene

rati

on


CDSS Characteristics: Highlights• Reviewed and classified 42 RCT-evaluated systems• Tremendous variation in decision-maker/context,

how recommendation delivered, staff needed to make system run, complexity of recommended actions– 45% targeted to clinician, 55% patient, 5% both– 62% based on national guidelines or literature– 69% “pushed” recommendation to decision maker– 43% collected data directly from the EMR

• 45% required data input intermediary (11% MD), 58% of time requiring clinical knowledge

– 26% required an output intermediary


CDSS Effectiveness Summary• Current data suggests CDSSs can improve the process of

care and perhaps clinical outcomes– most effective at preventive care reminders– modest at best for drug dosing and active care– generally not helpful for improving diagnosis except with trainees

• Findings limited by– methodological problems and design type of studies– insufficient appreciation of workflow component of CDSSs– insufficient appreciation of heterogeneity of systems

• Meta-regression on predictors of CDSS success using taxonomy, pending

• Bottom line: equivocal evidence, limited use


Outline






A Case...

• 51 y/o F prescribed camphorated tincture of opium• Given tincture of opium, died of morphine OD• Sons suing CVS for “pharmaceutical negligence and

wanton and reckless misconduct” for failing to “utilize, test and maintain” computer software capable of alerting employees to the dangers of confusing the two medications (Hartford Courant, Feb. 28, 2002)

• Was CVS negligent and reckless?– CVS could have used a rule-based system

• if Tincture of Opium => verify not Paregoric• if Tincture of Opium AND (dose > xyz) => verify dose


Prescription Error Checking

• Major errors can still slip through (Inst. Safe Medication Practices, 99)

– only 38% detect lethal cisplatin, vincristine doses– only 42% of systems linked between pharmacy and lab

• 87% did not detect excessive tobra dose for Cr level

• Drug-drug interaction detection– false positive rates in the 30% range currently– no commercial systems well accepted

• Alert fatigue is a real problem; other industries know “safety” systems can promote error


CPOE to Reduce Medication Errors• Computerized Physician Order Entry (CPOE)

– serious medication errors decreased 55% (Bates, 98)

• But only 1.7% of all errors result in patient harm (USP, 2002)

– nurses/pharmacists intercept half of ordering errors – absolute risk reduction less impressive

Error Type % of All Errors % Reduction

Physician Ordering 39

Dosing 23

Allergies 56

Transcription 12 84

Dispensing 68

Administration 38 59


CPOE Only Part of Solution

• CPOE does not address other “system inputs” to error

• Drew has CPOE– staff mistakenly entered a prescription for Gleevec into

CPOE for patient with meningitis (and no cancer) – nurses found error when comparing pharmacy records

and physician orders but did not remove the prescription from his medical record

– over the next 4 days, different nurses mistakenly administered Gleevec to patient (Los Angeles Times, 2/28)

• CPOE may promote error by decreasing staff vigilance


CPOE and Policy

• Leapfrog Group– influential group of businesses: deep pocket health

purchasers– hospitals get premium break if

• implement CPOE• dedicated ICU staff• evidence-based hospital referral

• Rush to implement CPOE– ahead/independent of EMR– may be complemented by bar coding, automated

dispensing carts, pharmacy decision support, etc.


Cedars-Sinai Experience• Patient Care Expert (PCX) CPOE

– developed ~$30 million system in-house– compares orders to recommendations, checks for allergies/drug

interactions, alerts to alternatives– expected 60-80% reduction in medical errors– rolled out October 2002

• A failure– CHF pt. did not receive prescribed pills, pt. got walker 3 days late,

baby got anesthetic day before circumcision– 2 mins. to order Vanco: sign on, search for screen, enter, justify,

verify; vs. 5 secs. previously– MDs delaying writing orders, writing them imprecisely– perception PCX was compromising patient safety

• Plug pulled mid-January 2003 (400 MD insurrection)• No plans to reintroduce CPOE until 2006 at the earliest


Outline






Fundamental Barrier

• Better quality care <-- better decision support• Better decision support <-- “smarter” systems

– “know” more about the patient, evidence, context

• “Smarter” systems <-- more richly coded data– for EMR: SNOMED, standard EMR structure– for knowledge: coding, structures for guidelines, RCTs…

• Coded data <-- Coding of data entry• Coding of data entry <-- Greater physician time• Greater physician time --> no play --> no gain


Implications

• Clear trade-off between physician coding effort and “smarter” decision support

• Don’t expect more decision support than coding supports– generally successful decision support

• preventive care: age, last mammogram, etc.• allergies: Yes/No on specific drugs• drug dosing: weight, height, creatinine, age

– generally unsuccessful decision support• diagnostic assistance• complicated therapies (e.g., management of hypertension)

• Unrealistic to think of CDSSs as improving evidence-based practice in general


The CVS Case

• Family recently settled for $1.8 million• Pharmacist had to take “intensive” error classes• State law changed, pharmacies must

– maintain error records– report errors to prescribing doctor, patient, or family

• No mention of having to use a computer system, etc.– error prevention needs a system approach; CPOE is but

one part of that


Summary on CDSSs• Equivocal evidence for improving quality

– limited by methodological and other shortcomings• Can reduce errors

– use in context of system approach– weigh costs and challenges against Absolute Risk Reduction

• Barriers in the short term– knowing which decisions to support (preventive care reminders)– data exchange among legacy systems– integrating all into a cohesive EMR solution– organizational challenges, incentivize for quality, etc. (3/16 class)

• Barriers in the long term– efficient coding: of what, by whom, when, why– clinical vocabulary with adequate semantic coverage– better reasoning methods


Teaching Points• CDSSs have great promise for improving

quality/reducing error– but promise essentially not yet realized

• CDSSs are heterogeneous in design and use– is a social/organizational intervention as much as a

technical one– need to improve methods for evaluating CDSSs

• Near-term difficulties in coding – should limit our sights on what to expect

• Need fundamental informatics research to crack coding barrier

March 2, 2004: I. Sim Clinical Decision Support Systems Medical Informatics Clinical Decision...

Documents

Transcript of March 2, 2004: I. Sim Clinical Decision Support Systems Medical Informatics Clinical Decision...