Dennis Ross-Degnan, ScD Harvard Medical School and Harvard Pilgrim Health Care Institute
-
Upload
nicodemus-illias -
Category
Documents
-
view
29 -
download
0
description
Transcript of Dennis Ross-Degnan, ScD Harvard Medical School and Harvard Pilgrim Health Care Institute
Evaluation Methods in Healthcare Quality Improvement:
Time Series Methods for Evaluating Quality Improvement Initiatives
Dennis Ross-Degnan, ScDHarvard Medical School and
Harvard Pilgrim Health Care Institute
2014 Academy for Healthcare Improvement Conference
Baltimore, MD, May 30, 2014
2
Overview
Rationale Study designs and inference Interrupted time series (ITS) design
Segmented regression analysis of ITS Data setup and models Estimating effects
Discussion of examples Strengthening ITS studies
3
RCTs: Gold Standard in Study Design but Rare in Natural or Quasi-Experiments
Randomized Controlled Trial
Intervention Group
Control GroupR
O1 X O2
Time
O1 O2
X=policy intervention Ot=Measurement at time t
4
RaPP: Analyzing a Group-Randomized Quality Improvement Intervention
Tailored intervention to improve use of antihypertensive and cholesterol medicines for primary prevention of CVD in Norway Educational outreach by pharmacist with audit+
feedback, EMR reminders (70 practices; 257 MDs) Controls receive passive dissemination of evidence-
based guidelines (69 practices; 244 MDs)
Outcomes measured monthly for eligible patients 1 year before & after intervention
Fretheim A, Oxman AD, Håvelsrud K et al. Rational Prescribing in Primary Care (RaPP): A cluster randomized trial of a tailored intervention. PLoS Medicine 2006.
Fretheim A, Soumerai SB, Zhang F, et al. Interrupted time series analysis yielded an effect estimate concordant with the cluster-randomized controlled trial result. Journal of Clinical Epidemiology 2013
5
Traditional Difference in Difference Analysis of RAPP RCT
RCT DiD: +9.0% (4.9%, 13.1 %)
6
“Quasi-Experimental” Design: Non-random Comparison Group
Non-random Comparison Group Design
Intervention Group
Comparison GroupR
O1 X O2
Time
O1 O2
X=policy intervention Ot=Measurement at time t
7
Strongest Quasi-Experimental Design:Interrupted Time Series (ITS)
Time
Stronger if includesComparison Group O1 O2 O3 O4 O5 O6
Experimental Group O1 O2 O3 X O4 O5 O6
X=policy intervention Ot=Measurement at time t
Time series: multiple measures of a single characteristic at equidistant time intervals
8
ITS Analysis of RaPP Study: Intervention Group Only
0
10
20
30
% a
dher
ence
to g
uide
lines
0 5 10 15 20 25Month
RAPP MDs
Adherence to Recommended Guidelines in RAPP Trial
9
ITS Logic and Parameters Estimated by Segmented Linear Regression
Adapted from Schneeweiss et al, Health Policy 2001
Baseline trend
Assumption: Baseline trend correctly reflects what would have happened without intervention
Immediate level change
Overall change at given time
before intervention after intervention Time
Outcome Intervention
Baseline level
Change from baseline trend
10
0
10
20
30
% a
dher
ence
to g
uide
lines
0 5 10 15 20 25Month
RAPP MDs Pre predicted Post predicted
Models adjusted for autocorrelation (AR1)
Fully specified modelAdherence to Recommended Guidelines in RAPP Trial
ITS Analysis of RaPP Study: Intervention Group Only
β0
β2
β1
β3
Yt = ß0 + ß1*time + ß2*intervention + ß3*time after interventionRCT DiD: +9.0% (4.9%, 13.1 %)ITS: +11.5% (9.5%, 13.5%)
Assumptions: 1.Linearity2.Normality3.Autocorrelatio
n structure
11
Does Adding RaPP Control Group Change Interpretation of Intervention Effects?
0
10
20
30
% a
dher
ence
to
gui
del
ines
0 5 10 15 20 25Month
RAPP MDs Control MDs
Adherence to Recommended Guidelines in RAPP Trial
RCT DiD: +9.0% (4.9%, 13.1 %)ITS: +11.5% (9.5%, 13.5%)ITS+comparison: +14.0% (8.6%, 19.4%)
Adapted from Soumerai et al, N Engl J Med 1987 12
0
2
4
6
8
Dec-79 Jun-80 Jan-81 Aug-81 Feb-82 Sep-82 Mar-83 Oct-83
Study Month
Avg
. # o
f pre
scri
ptio
ns
per
pat
ien
t 3 Rx per month cap begins
Cap replaced by $1 Copay
Effects of Reimbursement Caps Followed by Copayments in NH Medicaid
Multiple-Drug Recipients (n=860) All Other Patients (n=8002)
Adapted from Soumerai et al, N Engl J Med 1987 13
0
2
4
6
8
Dec-79 Jun-80 Jan-81 Aug-81 Feb-82 Sep-82 Mar-83 Oct-83
Study Month
Avg
. # o
f pre
scri
ptio
ns
per
pat
ien
t 3 Rx per month cap begins
Cap replaced by $1 Copay
Effects of Reimbursement Caps Followed by Copayments in NH Medicaid
Multiple-Drug Recipients (n=860) All Other Patients (n=8002)Back
14
Threats to Validity of ITS Design
Confounding: co-occurring intervention Selection: pre-intervention factors
affect inclusion (e.g., volunteers) Statistical regression: group(s)
selected because of baseline use Instrumentation: change in
measurement (ascertainment) History or maturation: external event
or natural process explains effect
15
Threats to Reliability of ITS Estimates
Data quality Short segments (few time points) Unstable data (high variability) Missing data or wild data points
Nature of population or process Changing denominators Low frequency (e.g., deaths) Near boundary (e.g., 0% or 100%) Non-linear trends
16
Summary
Advantages of ITS Intuitive visual display Direct estimate of effects Controls common threats to validity
Limitations of ITS Requires reasonably stable data Boundary problems Ideally 10+ data points per segment Sensitive to points near end of segment
17
Strengthening ITS Studies
Check data quality: Outliers, missing data, implausible data
Contrast multiple outcomes or groups: High-risk subgroups, different intensity
Account for intervention phase-in: Anticipatory effects, implementation time
Match on baseline values: Standardize or propensity match comparison groups
Test model assumptions: Normality of errors, linearity of segments
18
Discussion
Questions Additional topics
Data quality checking Sequential interventions Comparison groups Interpreting effects Selecting and matching study groups
19
Selected References1. Lagarde M. How to do (or not to do) …Assessing the impact of a policy change with routine
longitudinal data. Health Policy and Planning 2012;27:76-83.2. Linden A, Adams JL. Applying a propensity score-based weighting model to interrupted time
series data: improving causal inference in programme evaluation. J Evaluation in Clinical Practice 2010;1-8.
3. Schneeweiss S, Maclure M, Walker AM, Grootendorst P, Soumerai SB. On the evaluation of drug benefits policy changes with longitudinal claims data: the policy maker’s versus the clinician’s perspective. Health Policy 2001;55:97-109.
4. Serumag`a B, Ross-Degnan D, Avery A, Elliott RA, Majumdar SR, Zhang F, Soumerai SB. Effect of pay for performance on the management and outcomes of hypertension in the United KingdomL interrupted time series study. BMJ 2011;
5. Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Belmont, CA: Wadsworth, 2002.
6. Soumerai SB, Avorn J, Ross-Degnan D, Gortmaker S. Payment restrictions for prescription drugs under Medicaid. Effects on therapy, cost, and equity. N Engl J Med 1987;317:550-556.
7. Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. Journal of Clinical Pharmacy and Therapeutics 2002;27:299-309.
8. Wagner AK, Zhang F, Soumerai SB, Walker AM, Gurwitz JH, Glynn RJ, Ross-Degnan D. Benzodiazepine use and hip fractures in the elderly: Who is at greatest risk? Archives of Internal Medicine 2004;164:1567-1572.
9. Zhang F, Wagner AK, Soumerai SB, Ross-Degnan D. Methods for estimating confidence intervals in interrupted time series analyses of health interventions. J Clin Epidemiol 2009;62:143-148.
10. Zhang F, Wagner AK, Ross-Degnan D. Simulation-based power calculation for designing interrupted time series analyses of health policy interventions. J Clin Epidemiol. 2011; 64: 1252-61.
20
Good Internet Resources
STATA, SAS, R, SPSS http://www.ats.ucla.edu/stat/
SAS proc autoreg and Stata arima http://www.stata.com/statalist/archive/
2009-02/msg00140.html Correcting for Autocorrelation using
Stata http://www.polsci.wvu.edu/duval/ps602
/Notes/STATA/cocran-orcutt.htm Google!