1

EPI235: Epi Methods in HSRApril 5, 2005 L3

Evaluating Health Services using administrative data 2: Advanced Topics in Risk Adjustment (Dr. Schneeweiss)

Performance of risk adjustment tools in studies using administrative databases. Development of optimized risk adjustment using empirical weights and the application of propensity scores will conclude the topic. The application of instrumental variables will be briefly illustrated.

Background reading: •Schneeweiss S, Seeger J, Maclure M, Wang P, Avorn, J, Glynn RJ: Performance of comorbidity scores to control for confounding in epidemiologic studies using claims data. Am J Epidemiol, 2001;154:854-864. •Newhouse, JP, McClellan M: Econometrics in outcomes research. Ann Rev Public Health 1998;19:17-34.•Schneeweiss S, Wang PS, Avorn J, Glynn RJ: Improved Comorbidity Adjustment for Predicting Mortality in Medicare Populations. Health Services Research 2003; 38:1103-1120.

2

Risk adjustment or case mix adjustment

= controlling confounding (selection) bias

Patient characteristics

Provider Outcomes

Performance of risk adjustment tools

Developing optimal weights for comorbidity scores

Disease Risk Score vs. Exposure Propensity Scores

Instrumental variables

3

Data from British Columbia

141 161 BC residents 65 years or older 1-year baseline period (4/95 - 4/96)

6 comorbidity scoresUtilization measures

1-year follow-up (4/96-4/97)Mortality Hospitalizations (binary)Visits, Expenditures (continuos,

annualized)

4

Spearman Correlations at Baseline

# of # of# of # of elective emerg.

CDS-1 CDS-2 Deyo D'Hoore Romano Ghali Meds. Diags. hosp. hosp.CDS-2 0.653

Deyo 0.296 0.293

D'Hoore 0.298 0.306 0.587

Romano 0.305 0.301 0.892 0.594

Ghali 0.240 0.202 0.659 0.409 0.654

# of presription medications 0.646 0.779 0.343 0.349 0.351 0.275

# of diagnoses 0.257 0.327 0.287 0.319 0.289 0.251 0.422

# of elective hospitalizations 0.135 0.170 0.333 0.231 0.348 0.263 0.218 0.113

# of emergency hosp. 0.219 0.236 0.470 0.321 0.477 0.490 0.308 0.218 0.222

# of physician visits 0.298 0.371 0.321 0.341 0.332 0.273 0.470 0.711 0.288 0.313

Rx-based Dx-based

5

Performance: 1-year mortalityContinuos score

95%confidence

Score Model c limits

Age+gender 0.681 0.674 ;0.695

CDS-1 Age+gender+CDS-1 0.738 0.731 ;0.744

CDS-2 Age+gender+CDS-2 0.718 0.711 ;0.725

Deyo Age+gender+Deyo 0.768 0.762 ;0.775

D'Hoore Age+gender+D'Hoore 0.745 0.739 ;0.752

Romano Age+gender+Romano 0.771 0.764 ;0.777

Ghali Age+gender+Ghali 0.745 0.738 ;0.752

# Meds Age+gender+# of Meds 0.745 0.738 ;0.752

Schneeweiss et al. Am J Epidemiol 2001

6

Performance: Utilization Outcomes Binary outcomes (c-statistic) Continuos outcomes (R2)

Elective Emergency Long term care Physician ExpendituresScore Model * hospitalization hospitalization admissions visits Med. services

Demograph. Age 0.527 0.601 0.776 0.005 0.001Gender 0.528 0.515 0.553 0.000 0.001Age + gender 0.544 0.605 0.776 0.006 0.003

CDS-1 CDS-1 0.561 0.590 0.597 0.042 0.026+age+gender 0.575 0.637 0.792 0.046 0.028

CDS-2 CDS-2 0.579 0.605 0.601 0.048 0.032+age+gender 0.588 0.645 0.787 0.053 0.035

Deyo Deyo 0.580 0.601 0.644 0.077 0.045+age+gender 0.598 0.653 0.812 0.080 0.045

D'Hoore D'Hoore 0.578 0.597 0.669 0.068 0.039+age+gender 0.589 0.639 0.806 0.070 0.040

Romano Romano 0.585 0.604 0.649 0.081 0.039+age+gender 0.603 0.655 0.813 0.084 0.048

Ghali Ghali 0.552 0.577 0.622 0.061 0.028+age+gender 0.576 0.642 0.796 0.063 0.035

7

Performance: Simple Utilization Measures as Predictors

Binary outcomes (c-statistic) Continuos outcomes (R2)

Elective Emergency Long term care Physician ExpendituresMeasure Model hospitalization hospitalization admissions visits Med. services

# of meds # of meds 0.598 0.632 0.634 0.097 0.066+age+gender 0.609 0.668 0.798 0.101 0.070

# of diags # of diags 0.545 0.555 0.608 0.050 0.024+age+gender 0.564 0.619 0.794 0.053 0.027

# of emerg. hosp. # emerg. hosp. 0.555 0.593 0.651 0.060 0.033+age+gender 0.583 0.658 0.808 0.063 0.034

8

Performance: Combination of ICD-9 and Pharmacy Data

Binary outcomes (c-statistic) Continuos outcomes (R2)

Elective Emergency LTC Physician Expenditures forModel Mortality hospitalization hospitalization admissions visits med. services

Age 0.667 0.527 0.601 0.776 0.005 0.001

Gender 0.543 0.528 0.515 0.553 0.000 0.001

Age + gender 0.681 0.544 0.605 0.776 0.006 0.003

Age + gender + # meds + Deyo 0.781 0.624 0.680 0.818 0.131 0.084

Age + gender + # meds + D'Hoore 0.767 0.618 0.674 0.814 0.126 0.082

Age + gender + # meds + Romano 0.783 0.627 0.680 0.819 0.133 0.086

Age + gender + # meds + Ghali 0.770 0.616 0.678 0.807 0.125 0.081

9

Predictive validity and control for confounding

The apparent Relative Risk of an exposure-outcome relation can be expressed as

RRapp = f (RRtrue, Pr(C), Pr(E), OREC, RRCO).

Bias in % = {(RRapp - RRtrue)/RRtrue} * 100

10

-40

-20

0

20

40

60

80

100

120

0.1 1 10

OREC, Pr(E)=0.1

Bias

of R

REO

in %

CDS-1: c=0.721, p(C)=0.27CDS-2: c=0.715, p(C)=0.25Romano: c=0.757, p(C)=0.27Deyo: c=0.757, p(C)=0.26D'Hoore: c=0.717, p(C)=0.22

CDS-2

CDS-1

D'Hoore

Romano,Deyo

C

E Outcome

OREC

RRtrue = 1

RRCO

Schneeweiss et al. Am J Epidemiol 2001

11

Conclusions

ICD-9 based scores (Romano) perform better than Pharmacy based scores for a wide range of outcomes

A measure as simple as the number of distinct prescription medications out-performs some scores

We recommend to combine Romano with the number of distinct prescription medications

Predictive performance translates into the capacity to control bias.

12

Generalizability?

Caution when generalizing performance to other populations especially to patients with specific conditions

However, ranking of scores may be generalizable to a larger extent

13

British Columbia(cardio-vasc.*)n = 141,161

New Jersey / PAAD(cardio-vasc.*)n = 108,247

New Jersey / PAAD(total)

n = 235,881

Pennsylvania/PACE(total)

n = 230,91395%

confidence95%

confidence95%

confidence95%

confidenceScore Model c limits c limits c limits c limits

Age + sex 0.681 0.674 ; 0.695 0.664 0.658 ; 0.671 0.658 0.654 ; 0.662 0.669 0.665 ; 0.673

CDS-1 Age + sex + CDS-1 0.738 0.731 ; 0.744 0.689 0.682 ; 0.695 0.703 0.699 ; 0.707 0.695 0.691 ; 0.699

CDS-2 Age + sex + CDS-2 0.718 0.711 ; 0.725 0.677 0.670 ; 0.683 0.690 0.686 ; 0.694 0.683 0.679 ; 0.687

D’Hoore Age + sex + D’Hoore 0.745 0.739 ; 0.752 0.745 0.739 ; 0.751 0.760 0.757 ; 0.764 0.747 0.743 ; 0.750

Ghali Age + sex + Ghali 0.745 0.738 ; 0.752 0.738 0.732 ; 0.744 0.739 0.735 ; 0.743 0.733 0.729 ; 0.737

Deyo Age + sex + Deyo 0.768 0.762 ; 0.775 0.753 0.748 ; 0.759 0.768 0.764 ; 0.772 0.757 0.753 ; 0.761

Romano Age + sex + Romano 0.771 0.764 ; 0.777 0.754 0.748 ; 0.760 0.771 0.767 ; 0.775 0.757 0.754 ; 0.761

# of Rx ** Age +sex + number of drugsused in past year

0.745 0.738 ; 0.752 0.701 0.695 ; 0.707 0.713 0.709 ; 0.717 0.701 0.697 ; 0.705

Romano + # of Rx Age + sex + Romano +number of drugs used in pastyear

0.783 0.778 ; 0.789 0.756 0.750 ; 0.761 0.775 0.771 ; 0.779 0.760 0.756 ; 0.764

Rx-basedscores

Dx-basedscores

Ranking is preserved across 4 populations of elderly beneficiaries

Schneeweiss et al. JGIM 2004

14

Exercise:

You are an assistant to the Chief Medical Officer of a physician group. 50% of

the group’s revenue in primary care comes from capitation plans. The revenue

is pooled for the group and than distributed based on the number of patients

each physician treated last month.

After several complaints your CMO decided to pay physicians not only by the

number of patients but also based on demographics and morbidity of patients.

She asked you to come up with a feasible and not too expensive solution.

The group has electronic administrative databases including diagnostic

information starting 5 years ago.

1. Define your goal

2. Find a cost-effective strategy to reach your goal

3. Propose a design to test the performance of your new system

15

Can we Improve Scores?

To what extend can the performance of comorbidity scores be improved using specific weights for elderly populations using claims data?

16

Subjects

235,881 NJ Medicare enrollees with drug coverage through Medicaid or PAAD.

Continuous beneficiary throughout 1994

17

Two Comorbidity Indicators:1-year baseline period (1/94 - 12/94)

Charlson score (Romano’s algorithm)Chronic Disease Score (von Korff’s algorithm)

1-year mortality during follow-up year (1/95-12/95)

Study Outcome:

18

Measure of Improved Performance

Performance of prediction: Area under the ROC curve Percent improvement of prediction beyond chance

1 - Specificity

Sensitivity

AUC = .7

AUC = .8

AUC = .5

% improvement:

(0.8-0.7) / (0.7-0.5) = 50%

19

ResultsSelected conditions of the Charlson score and weights derived from New JerseyMedicare data (N = 235,881).

Conditions

NJpreva-lence

Charlsonweights

NJ oddsratios 95% CI

NewMedicareweights*

Myocardial infarct 4.3 % 1 1.23 (1.15-1.31) 1Congestive heart failure 15.1 % 1 2.09 (2.01-2.17) 2Peripheral vascular disease 13.3 % 1 1.55 (1.49-1.61) 1Cerebrovascular disease 11.4 % 1 1.42 (1.36-1.48) 1Dementia 6.2 % 1 2.16 (2.06-2.27) 3Chronic pulmonary disease 12.4 % 1 1.66 (1.59-1.73) 2Connective tissue disease 2.2 % 1 1.09 (0.98-1.21) 0Ulcer disease 3.4 % 1 1.03 (0.96-1.11) 0Mild liver disease 0.3 % 1 1.73 (1.41-2.12) 2Diabetes 12.0 % 1 1.37 (1.31-1.44) 1Any tumor 5.8 % 2 1.85 (1.75-1.95) 2Metastatic solid tumor 1.7 % 6 5.94 (5.50-6.40) 6AIDS 0.1 % 6 3.26 (2.13-4.98) 4* A 35% increase in risk of dying is reflected in a one point increase in weights.

20

ResultsSelected conditions in the Chronic Disease Score (CDS) and weights derived fromNew Jersey Medicare data (N= 235,881).

Conditions and medicationsNJ

preva-lence

CDSweights

NJoddsratios

NewMedicareweights

Heart disease: (1) Anti-coagulants, hemostatics, one class 42.3 % 3 1.45 1 (2) Cardiac agents, ACE inhibitors, two classes 16.3 % 4 2.53 3 (3) Loop diuretics three classes 2.3 % 5 3.14 4Asthma, rheumatism: Glucocorticoids 8.7 % 3 1.32 1Rheumatoid arthritis: Gold salts 0.1 % 3 0.80 -1Hypertension: (1) Antihypertensives (except ACEI) or calcium class (1) 18.4 % 2 0.95 0

channel blockers, (2) Beta blockers, diuretics class (2) not (1) 10.4 % 1 1.03 0Asthma, rhinitis: Cromolyn 0.6 % 2 0.54 -2Acne: (1) Antiacne tretinoin, (2) topical macrolides 0.2 % 1 0.49 -2High cholesterol: Antilipidemics 10.0 % 1 0.59 -2

21

ResultsImprovement in performance of comorbidity scores by using Medicare weights inthe development and validation samples.

Population sample Score* Originalweights

New weightsPerformanceimprovement

in %Development sample(New Jersey)

CDS 0.703(0.699;0.707)

0.727(0.723;0.731)

35 %

Charlson 0.771(0.767;0.775)

0.780(0.776;0.783)

7.4 %

Validation sample(Pennsylvania)

CDS 0.695(0.691;0.699)

0.715(0.711;0.719)

43 %

Charlson 0.757(0.754;0.761)

0.765(0.762;0.769)

8.3 %

Schneeweiss et al. Health. Serv. Res., 2003

22

The Next Level?

Individualized „Summary Risk Scores“ (Ray e.g. 2002)

Multivariate Confounder Scores Miettinen 1976 Pike, Anderson 1979 Cook, Goldman 1989

= Disease Risk Score (DRS)

23

Disease Risk Score estimation: Objective: more efficient control for confounding / risk adjustment

2 Models: Risk core model

Outcome model

1. Risk Score Model:

Logit (p(death)) = a + b0*TRT + b1*age + b2*sex + … + e;

Disease risk score = the predicted value for outcome [0,1] for each individual

independent of exposure.

Two strategies: a) estimate disease risk score in the unexposed

b) estimate disease risk score adjusted for exposure

2. Outcome Model:

Logit (p(death)) = α + β1*DRS + β2*TRT + e

Alternatives strategies to use propensity scores (DRS):

Continuous DRS

Quintile indicators for DRS

Matching by DRS

24

Exposure Propensity Score estimation: Objective: “better” control for confounding / risk adjustment

2 Models: Treatment model

Outcome model

1. Treatment Model:

Logit (p(TRT)) = a + b1*age + b2*sex + … + e;

Exposure Propensity score (EPS) = the predicted value for treatment [0,1] for

each individual.

Objective of treatment model? How to evaluate whether objective was

achieved?

2. Outcome Model:

Logit (p(death)) = α + β1*EPS + β2*TRT + e

Alternatives strategies to use propensity scores (PS):

Continuous EPS

Quintile indicators for EPS

Matching by EPS

25

Reasons for Data Reduction???

1.

2.

3.

26

Instrumental Variable estimation

Goal: Adjusting for measured and UNmeasured covariates in non-experimental

research.

IV Patient characteristics

Provider/ Treatment

Outcomes

Two important conditions:

1) IV not associated with confounders

2) IV not associated with outcome other than through the actual exposure

IV estimation is similar to randomization.

IV estimates are marginal effect estimates (like RCT estimates) not conditional

effect estimates (like regression estimates)

27

IV estimation

Good IVs are rare Geographic variation Distance to (specialty) provider (McClellan JAMA) Turning 65 Physician preference (Brookhart, 2006; Schneeweiss 2007) ?

28

Several outcome models:

Logit (p(death)) = a + b1*age + b2*sex + b3*comorbidity + b4*provider/trt + e

Logit (p(death)) = a + b1*age + b2*sex + b3*propensity score + b4*provider/trt + e

Logit (p(death)) = a + b1*IV + e

Logit (p(death)) = a + b1 *Randomization Status + e

29

Overview

StratificationMatchingRegression modelsDRS (RA tools)EPSIVRandomizationSensitivity AnalysisCross-over design

Can adjust UNmeasured confounders? Improves efficiency?