1 Evaluating Health Information Technology: Putting Theory Into Practice Eric Poon, MD MPH Clinical...
-
Upload
gerard-quinn -
Category
Documents
-
view
214 -
download
0
Transcript of 1 Evaluating Health Information Technology: Putting Theory Into Practice Eric Poon, MD MPH Clinical...
1
Evaluating Health Information Technology: Evaluating Health Information Technology: Putting Theory Into PracticePutting Theory Into Practice
Eric Poon, MD MPHEric Poon, MD MPHClinical Informatics Research and Development,
Partners Information Systems
David F. Lobach, MD, PhD, MSDavid F. Lobach, MD, PhD, MS
Division of Clinical Informatics Division of Clinical Informatics
Department of Community and Family MedicineDepartment of Community and Family Medicine
Duke University Medical Center, Durham, North CarolinaDuke University Medical Center, Durham, North Carolina
AHRQ’s National Resource Center for Health Information TechnologyAHRQ’s National Resource Center for Health Information Technology
Annual Meeting June 2005Annual Meeting June 2005
2
OutlineOutline
Overview of Evaluating HITOverview of Evaluating HIT Why evaluate?Why evaluate? General Approach to EvaluationGeneral Approach to Evaluation Choosing Evaluation MeasuresChoosing Evaluation Measures Study Design TypesStudy Design Types Analytical issues in HIT evaluationsAnalytical issues in HIT evaluations
Evaluation in the ‘real world’Evaluation in the ‘real world’ Duke University Medical Center Duke University Medical Center
3
Why Measure Impact of HIT?Why Measure Impact of HIT?
Impact of HIT often hard to predict Impact of HIT often hard to predict Many “slam dunks” go awryMany “slam dunks” go awry You can’t manage/improve what isn’t measuredYou can’t manage/improve what isn’t measured
Understand how to clear barriers to effective Understand how to clear barriers to effective implementationimplementation Understand what works and what doesn’tUnderstand what works and what doesn’t Invent the wheel only onceInvent the wheel only once
Justify enormous investmentsJustify enormous investments Return on investmentReturn on investment Allow other institutions to make tradeoffs intelligentlyAllow other institutions to make tradeoffs intelligently
Use results to win over late adoptersUse results to win over late adopters
4
General Approach to Evaluating HITGeneral Approach to Evaluating HIT
Understand your interventionUnderstand your intervention
Formulate questions to answerFormulate questions to answer
Select and define measuresSelect and define measures
Pick the study designPick the study design
Data analysisData analysis
5
Getting Started: Get to know your Getting Started: Get to know your interventionintervention
What problem(s) is it trying to solve?What problem(s) is it trying to solve? Think about intermediate processesThink about intermediate processes
Identify potential barriers to successful Identify potential barriers to successful implementation:implementation: Managerial barriersManagerial barriers End-user behavioral barriersEnd-user behavioral barriers
Understand how your peers around the Understand how your peers around the country are addressing (or not) the same country are addressing (or not) the same issues.issues.
6
Formulating QuestionsFormulating Questions
Likely questions:Likely questions: Does the HIT work?Does the HIT work? What would have made it work better?What would have made it work better? What would the next set of designers/implementors What would the next set of designers/implementors
like to know?like to know? Has this question been fully answered before?Has this question been fully answered before?
Don’t reinvent the wheel! (not a big concern)Don’t reinvent the wheel! (not a big concern) What impact would the answer have?What impact would the answer have?
PeersPeers Policy makersPolicy makers
7
Array of MeasuresArray of Measures
Quality and SafetyQuality and Safety Clinical OutcomesClinical Outcomes Clinical ProcessesClinical Processes
KnowledgeKnowledge PatientPatient ProviderProvider
Satisfaction & Satisfaction & AttitudesAttitudes PatientPatient ProviderProvider
Resource utilizationResource utilization Costs and chargesCosts and charges LOSLOS Employee Employee
time/workflowtime/workflow
Lessons learnedLessons learned
8
Choosing Study MeasuresChoosing Study Measures Clinical vs Process MeasuresClinical vs Process Measures
Clinical outcomes (e.g. mortality) desirableClinical outcomes (e.g. mortality) desirable Justifiable to measure process outcomes (e.g. door to abx time) Justifiable to measure process outcomes (e.g. door to abx time)
if relationship between outcome and process already if relationship between outcome and process already demonstrateddemonstrated
Will outcomes be impacted by the intervention?Will outcomes be impacted by the intervention? Will impact on outcomes be detectable during the study Will impact on outcomes be detectable during the study
period?period? ? Rare events, e.g. adverse outcomes? Rare events, e.g. adverse outcomes ? Colon cancer screening? Colon cancer screening
What resources do you have?What resources do you have? Don’t bit off more than what you can chew.Don’t bit off more than what you can chew.
9
Selecting Study TypesSelecting Study Types
Commonly used study types:Commonly used study types: Optimal design: Randomized Controlled TrialsOptimal design: Randomized Controlled Trials
Factorial DesignFactorial Design Before-and-after time series TrialsBefore-and-after time series Trials
Main study design issues:Main study design issues: Secular Trend: Can a simultaneous control group Secular Trend: Can a simultaneous control group
be established?be established? Confounding: Can you randomly assign Confounding: Can you randomly assign
individuals to study groups?individuals to study groups? Study design often influenced by Study design often influenced by
implementation planimplementation plan Need to respect operational needs, but often Need to respect operational needs, but often
there is room for creative designsthere is room for creative designs
10
Randomization Nuts and BoltsRandomization Nuts and Bolts
Justifiable to have a control arm as long as Justifiable to have a control arm as long as benefit not already demonstrated (usual care)benefit not already demonstrated (usual care)
Want to choose a truly random variable Want to choose a truly random variable Not day of the weekNot day of the week
Consideration: Stratified randomizationConsideration: Stratified randomization Ensures that intervention and control group are Ensures that intervention and control group are
similar on important characteristics (e.g. baseline similar on important characteristics (e.g. baseline computer literacy)computer literacy)
Strongest possible interventionStrongest possible intervention
11
Randomization Unit:Randomization Unit:How to Decide?How to Decide?
Small units (patients) vs. Large units (practices Small units (patients) vs. Large units (practices wards)wards) Contamination across randomization unitsContamination across randomization units If risk of contamination is significant, consider If risk of contamination is significant, consider
larger unitslarger units Effect contamination-can underestimate impactEffect contamination-can underestimate impact
However, if you see a difference, impact is presentHowever, if you see a difference, impact is present
Randomization by patient generally undesirableRandomization by patient generally undesirable ContaminationContamination Ethical concernEthical concern
12
Randomization Schemes:Randomization Schemes:Simple RCTSimple RCT
Burn-in periodBurn-in period Give target population time to get used to new Give target population time to get used to new
intervention intervention Data not used in final analysisData not used in final analysis
XX Clinics
Baseline Period
Baseline Data Collection Data Collection for RCT
No Intervention
Intervention Period
3 month burn-in period
Intervention Deployed
Intervention
arm
Control
arm
Control arm gets intervention
Post- Intervention
Period
13
Randomization schemes: Randomization schemes: Factorial DesignFactorial Design
May be used to May be used to concurrently evaluate concurrently evaluate more than one more than one intervention:intervention: Assess interventions Assess interventions
independently and in independently and in combinationcombination
Loss of statistical powerLoss of statistical power
Usually not practical for Usually not practical for more than 2 interventionsmore than 2 interventions
Control (no interventions)
A
B
A+B
14
Randomization Schemes:Randomization Schemes:Staggered DeploymentStaggered Deployment
Advantages of staggeringAdvantages of staggering Easier for user education and trainingEasier for user education and training Can fix IT problems up frontCan fix IT problems up front
Need to account for secular trend and baseline differencesNeed to account for secular trend and baseline differences Time variable in regression analysisTime variable in regression analysis Control for practice characteristicsControl for practice characteristics
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2Practice 1
Practice 2
Practice 3
Practice 4
Practice 5
Practice 6
Practice 7
Practice 8
2006 2007 2008
Intervention Group
Control Group
Intervention Group
Control Group
Intervention Group
Control Group
Intervention Group
Control Group
15
Inherent Limitations of RCTs in Inherent Limitations of RCTs in InformaticsInformatics
Blinding is seldom possibleBlinding is seldom possible
Effect on documentation vs. clinical actionEffect on documentation vs. clinical action
People always question generalizabilityPeople always question generalizability Success is highly implementation independentSuccess is highly implementation independent Efficacy-effectiveness gap: ‘Invented here’ Efficacy-effectiveness gap: ‘Invented here’
effecteffect
16
Mitigating the Limitations of Mitigating the Limitations of Before-and-After Study DesignsBefore-and-After Study Designs
Before-and-after trial common in informaticsBefore-and-after trial common in informatics Concurrent randomization is hardConcurrent randomization is hard
Don’t lose the opportunity to collect baseline Don’t lose the opportunity to collect baseline data!data!
Leave the time gap between before and after Leave the time gap between before and after trends relatively shorttrends relatively short
Look for secular trend in statistical analysis and Look for secular trend in statistical analysis and adjust for it if presentadjust for it if present
17
Common Pitfalls with Data CollectionCommon Pitfalls with Data Collection
Measures you define and collect on your ownMeasures you define and collect on your own Pilot data collection and refine definition Pilot data collection and refine definition earlyearly Ask yourself Ask yourself earlyearly whether data your collect measure whether data your collect measure
what you intended to measure.what you intended to measure. Measures others defined but you collect on your Measures others defined but you collect on your
ownown Do you need to adapt other people’s instruments?Do you need to adapt other people’s instruments?
Measures others define and collect for youMeasures others define and collect for you Understand nuisances and limitations, particular with Understand nuisances and limitations, particular with
administrative data.administrative data.
18
Electronic Data Abstraction: Electronic Data Abstraction: There’s no free lunch!There’s no free lunch!
Convenient and time-saving, but…Convenient and time-saving, but… Some chart review (selected) to get Some chart review (selected) to get
information not available electronicallyinformation not available electronically Get ready for surprisesGet ready for surprises
Documentation effect of EMRsDocumentation effect of EMRs
19
Data Collection Issue: Data Collection Issue: Baseline DifferencesBaseline Differences
Randomization schemes Randomization schemes often lead to imbalance often lead to imbalance between intervention and between intervention and control arms:control arms: Need to collect baseline Need to collect baseline
data and adjust for data and adjust for baseline differences baseline differences
Interaction term ( Time * Interaction term ( Time * Allocation Arm) gives Allocation Arm) gives effect for intervention in effect for intervention in regression analysisregression analysis
20
Data Collection Issue: Data Collection Issue: Completeness of FollowupCompleteness of Followup
The higher the better:The higher the better: Over 90%Over 90% 80-90%80-90% Less than 80%Less than 80%
Intention to treat analysisIntention to treat analysis In an RCT, should analyze outcomes In an RCT, should analyze outcomes
according to the original randomization according to the original randomization assignmentassignment
21
A Common Analytical Issue A Common Analytical Issue The Clustering EffectThe Clustering Effect
Occurs when your observations are not Occurs when your observations are not independent:independent: Example: Each physician treats multiple patients:Example: Each physician treats multiple patients:
May need to increase sample size to account for May need to increase sample size to account for loss of power.loss of power.
Intervention Group Control Group
Physicians
Patient -> Outcome assessed
22
Looking at Usage DataLooking at Usage Data
Great way to tell how well the intervention Great way to tell how well the intervention is goingis going Target your trouble-shooting effortsTarget your trouble-shooting efforts
In terms of evaluating HIT:In terms of evaluating HIT: Correlate usage to implementation/training Correlate usage to implementation/training
strategystrategy Correlate usage to stakeholder characteristicsCorrelate usage to stakeholder characteristics Correlate usage to improved outcomeCorrelate usage to improved outcome
23
Studies on Workflow and UsabilityStudies on Workflow and Usability
How to make observations?How to make observations? Direct observationsDirect observations Stimulated observationsStimulated observations
Random paging methodRandom paging method Subjects must be motivated and cooperativeSubjects must be motivated and cooperative
Usability LabUsability Lab What to look for?What to look for?
Time to accomplish specific tasks:Time to accomplish specific tasks: Need to pre-classify activitiesNeed to pre-classify activities Handheld/Tablet PC tools may be very helpfulHandheld/Tablet PC tools may be very helpful
Workflow analysisWorkflow analysis Asking users to ‘think aloud’Asking users to ‘think aloud’
Unintended consequences of HITUnintended consequences of HIT
24
Cost Benefit AnalysisCost Benefit Analysis
Do the benefits of the technology justify the costs?Do the benefits of the technology justify the costs? Monetary benefits – Monetary costsMonetary benefits – Monetary costs Important in the policy realmImportant in the policy realm
Need to specify perspectiveNeed to specify perspective OrganizationalOrganizational SocietalSocietal
Cost analysis more straight forwardCost analysis more straight forward Prospective data collection preferredProspective data collection preferred Discounting: a dollar spent today worth more than a dollar 10 years from Discounting: a dollar spent today worth more than a dollar 10 years from
nownow Benefits analysis more controversialBenefits analysis more controversial
Cost of illness averted: medical costs, productivity for patient Cost of illness averted: medical costs, productivity for patient What is the cost of suffering due to preventable adverse events?What is the cost of suffering due to preventable adverse events? What is the cost of a life?What is the cost of a life?
25
Using Surveys – Stay Tuned!Using Surveys – Stay Tuned!
Survey of user believes, attitude and Survey of user believes, attitude and behaviorsbehaviors Response rate – responder bias: Aim for Response rate – responder bias: Aim for
response rate > 50-60%response rate > 50-60% Keep the survey conciseKeep the survey concise Pilot survey for readability and clarityPilot survey for readability and clarity Need formal validation if you want plan to Need formal validation if you want plan to
develop a scale/summary scoredevelop a scale/summary score
26
Qualitative Methodologies – Qualitative Methodologies – Don’t touch that dial!Don’t touch that dial!
Major techniquesMajor techniques Direct observationsDirect observations Semi-structured interviewsSemi-structured interviews Focus groupsFocus groups
Adds richness to the evaluationAdds richness to the evaluation Explains successes and failures. Generate Lessons learnedExplains successes and failures. Generate Lessons learned Captures the unexpectedCaptures the unexpected Great for forming hypothesesGreat for forming hypotheses People love to hear storiesPeople love to hear stories
Data analysisData analysis Goal is to make sense of your observationsGoal is to make sense of your observations Iterative & interactiveIterative & interactive
27
Concluding RemarksConcluding Remarks
Don’t bite off more than what you can chewDon’t bite off more than what you can chew Pick a few study outcomes and study them well. Pick a few study outcomes and study them well.
It’s a practical worldIt’s a practical world Balancing operational and research needs is Balancing operational and research needs is
always a challenge.always a challenge. Life (data collection) is like a box of Life (data collection) is like a box of
chocolates…chocolates… You don’t know what you’re going to get until you You don’t know what you’re going to get until you
look, so look early!look, so look early!
28
Thank youThank you
Eric Poon, MD MPHEric Poon, MD MPH Email: Email: [email protected]@partners.org
AcknowledgementsAcknowledgements Davis Bu, MD MADavis Bu, MD MA
CITL, Partners HealthcareCITL, Partners Healthcare David Bates, MD MScDavid Bates, MD MSc
Chief, Div of General Medicine, Brigham and Chief, Div of General Medicine, Brigham and Women’s HospitalWomen’s Hospital