Covariate Selection and Balance in Propensity Score Methods · Covariate Selection and Balance in...

Covariate Selection and Balance in

Propensity Score Methods

M Sanni Ali

University Medical Center Utrecht, the Netherlands

Outline

Confounding

Propensity Score (PS) Methods

Covariate Selection

Balance Diagnostics and their Applications

Balance Assessment

PS Methods in Time-varying Treatment

Reporting of PS Analysis

Confounding

Non-intermediate common cause of exposure

and outcome

Unblocked backdoor path in causal diagrams

Observed, unobserved, unknown

Propensity Score Methods

Propensity Score

- Summary Score – Variables included?- Balancing Score – Covariate balance?

Often derived using logistic regression

- Selection of variables and forms

Ensemble methods

- Automated – Interactions/polynomials

Matching, stratification, weighting, and covariate adjustment

Rubin & Rosenbaum 1983; Westreich et al 2010; Lee 2010

Advantages of PS

Confidence on the causal inference

Transparency/Easy to communicate

Design a study separate from analysis

Summary Scores – rare outcome

Covariate Selection

Clinical knowledge - Sufficient?

Variable with Instrumental Variable like properties

- Exacerbate the imbalance in unmeasured confounders = Amplify the bias

When could bias be amplified

- Unmeasured Confounding – Strong!!!

- Strong IV !!!- Linear models

Balance diagnostics are powerful tools

- Absolute standardized difference is robust

Bhattacharya & Vogt 2007; Pearl 2010&2011; Myers et al 2011; Austin 2009;

Belitser et al 2011; M S Ali et al 2014

A simulation study using binary covariates, treatment, and outcome.

In different scenarios, confounding variables, risk factors, instrumental variables, their interaction and square terms were considered.

M S Ali et al. 2014

Covariate Selection

A

X9

Y

X7

X8

X6

X3

X1, X2 , X5

X4

Covariate Balance

PS model versus covariate sets (balance)

Four sets of covariates

- Full PS Model

- True PS Model

- Outcome PS model (Prognostic score)

- Confounder PS model

Main terms versus Main + Interaction/squares

Covariate Selection

9

Description of the Different Propensity Score Models

Covariate Selection

Treatment effects estimates (risk ratios) were derived using Poisson models.

PS model selection was made based on the balance achieved on different sets of covariates, their interaction/square terms.

Covariate balance was assessed using the absolute standardized difference.

Covariate sets in balance assessment were compared with respect to bias and precision of the treatment-outcome relation as well as the PS model selected.

Covariate Selection - Balance

11

Data was unmatched* and matched using Full PS model (PS_2), True PS model (PS_18), Risk Factor PS model (PS_25),

Confounder PS model (PS_11), Confounder PS model omitting confounder, X4 (PS_15), Confounder PS model omitting confounder,

X4, but risk factors were included (PS_29), Confounder PS model omitting confounder, X4, but instrumental variables were included

(PS_22), Confounder PS model omitting confounder, X4, but risk factors and instrumental variables were included (PS_8).

Table 1. Balance of Covariates Measured Using Standardized Difference When Different Sets of Covariates Were Included in the PS Model

Covariate Selection - Balance

12






Covariate Selection: balance

13






ModelAll Covariates

were Independent

RR (IQR)

X4 and X5 were

Correlated

RR (IQR)

X4 and X7 were

Correlated

RR (IQR)

Crude 0.93 0.15 0.89 0.17 0.91 0.14

Full PS Model 1.73 0.40 1.78 0.68 1.72 0.59

True PS Model 1.71 0.47 1.73 0.63 1.67 0.58

Outcome PS Model 1.71 0.39 1.77 0.48 1.72 0.51

Conf PS Model 1.77 0.40 1.79 0.56 1.71 0.50

Omit.Conf PS Model 1* 1.55 0.34 1.57 0.40 1.46 0.35

Omit.Conf PS Model 2** 1.54 0.32 1.59 0.39 1.45 0.34

Omit.Conf PS Model 3† 1.50 0.36 1.50 0.50 1.51 0.47

Omit.Conf PS Model 4†† 1.50 0.34 1.55 0.52 1.52 0.45

Table 4. Median (Interquartile Range, IQR) of Estimated Treatment Effect Using Different PS Models in Different Scenarios

(True RR=1.75)

Covariate Selection

15

PS matching improved balance of measured covariates included in the PS model

It exacerbated the imbalance in the unmeasured covariate that was unrelated to measured covariates

In choosing covariates for a PS model, the pattern of association among covariates has substantial impact on other covariates’ balance and the bias of the treatment effect estimate.

Table 5. Median (IQR) Treatment Effect When Propensity Score Model was Chosen Based on Balance Calculations on Different Sets of Covariates (True RR =1.75)

Calliper All

CovariatesConfounding

Factors

Confounding and Risk Factors

Confounding and Treatment Related Factors

0.05 1.51 0.42 1.77 0.64 1.75 0.62 1.48 0.43

0.10 1.50 0.43 1.75 0.63 1.75 0.58 1.48 0.42

0.15 1.50 0.44 1.73 0.67 1.76 0.65 1.48 0.43

0.20 1.48 0.42 1.75 0.62 1.75 0.60 1.46 0.43

0.25 1.46 0.43 1.75 0.61 1.72 0.58 1.43 0.42

0.30 1.47 0.43 1.74 0.62 1.72 0.60 1.43 0.42

0.35 1.44 0.41 1.69 0.61 1.69 0.57 1.41 0.39

0.40 1.40 0.41 1.68 0.59 1.67 0.61 1.37 0.39

0.50 1.38 0.36 1.68 0.54 1.63 0.50 1.35 0.35

0.60 1.33 0.37 1.58 0.51 1.55 0.52 1.31 0.34

*Balance was assessed on main terms as well as interaction/square terms

Table 6. Median (IQR) Treatment Effect When Propensity Score Model was Chosen Based on Balance Calculations on Different Sets of Covariates (True RR =1.75)

Calliper All Covariates

Confounding

Factors

Confounding and

Risk Factors

Confounding and

Treatment Related

Factors

0.05 1.73 0.59 1.74 0.47 1.73 0.51 1.77 0.67

0.10 1.76 0.64 1.78 0.52 1.75 0.54 1.75 0.62

0.15 1.72 0.68 1.75 0.50 1.73 0.54 1.75 0.63

0.20 1.74 0.60 1.74 0.50 1.74 0.45 1.74 0.61

0.25 1.70 0.65 1.74 0.52 1.71 0.50 1.72 0.61

0.30 1.71 0.59 1.72 0.53 1.72 0.53 1.71 0.58

0.35 1.67 0.56 1.70 0.48 1.68 0.47 1.68 0.58

0.40 1.67 0.60 1.70 0.49 1.68 0.48 1.66 0.56

0.50 1.60 0.53 1.65 0.49 1.62 0.48 1.61 0.55

0.60 1.54 0.48 1.65 0.45 1.64 0.46 1.53 0.50

*Balance was assessed only main terms not interaction/square terms

Table 7. Median Number of Subjects Included When Propensity Score Model was Chosen Based on Balance Calculations on Different Sets of Covariates (True RR =1.75, n=2000)

*Balance was assessed main and interaction/square terms


Confounding

Factors

Confounding and

Risk Factors

Confounding and

Treatment Related

Factors

0.05 1388 1388 1388 1388

0.10 1390 1389 1389 1390

0.15 1391 1391 1391 1391

0.20 1393 1392 1392 1393

0.25 1394 1394 1394 1394

0.30 1397 1396 1396 1397

0.35 1397 1398 1398 1398

0.40 1401 1401 1401 1401

0.50 1408 1408 1406 1408

0.60 1415 1417 1415 1415

Table 8. Median Number of Subject Included When Propensity Score Model was Chosen Based on Balance Calculations on Different Sets of Covariates (True RR =1.75, n=2000)

*Balance was assessed only main terms not interaction/square terms


Confounding

Factors

Confounding and

Risk Factors

Confounding and

Treatment Related

Factors

0.05 1388 1481 1479 1388

0.10 1390 1481 1479 1389

0.15 1392 1479 1478 1391

0.20 1393 1479 1479 1392

0.25 1393 1479 1478 1393

0.30 1396 1480 1480 1396

0.35 1398 1482 1481 1397

0.40 1401 1484 1484 1401

0.50 1407 1488.5 1489 1407

0.60 1415 1496 1496 1415

PS Model Selection

When PS model was chosen based on balance calculation on confounding variables, the confounder PS is often selected followed by the Risk factor PS models.

When balance calculation involves all covariates or treatment related covariates, the full PS and true PS model are often selected compared to the confounder PS and Risk factor PS models.

PS Model Selection

In selecting PS model based on covariate balance, the choice of covariates, interaction/square terms in balance calculation has substantial impact on bias and precision of the treatment effect.

PS model selection based on the balance achieved on confounding variables, risk factors and important interaction terms among confounders and risk factors is optimal approach.

Balance DiagnosticsQuantitative Falsification of Instrumental Variable Assumptions

Instrumental Variable Methods

three Basic IV assumption

- Associated with Exposure

- Independent of Confounders

- Independent of outcome except through exposure

Association of IV with observed confounders could be

quantified with balance metrics

M S Ali et al 2014

IV Methods and assumptions

PS and time-varying treatment

In the presence of treatment switching or

channeling - “Intension to treat analysis”?

Time-varying Cox model or

Time-varying propensity score or

Inverse probability weighting or others

Time-varying treatment

Example

Associations Between Current SSRI Use and the Risk of Hip Fracture

Study using two EU databases (Mondriaan and BIFAP)

A cohort of patients with a first prescription for Antidepressant (SSRI or tricyclic AD, TCA) period 2001-2009

Data sources (GP databases) :- the Dutch Mondriaan - the Spanish BIFAP

M S Ali et al 2015


HFSSRItSSRIt-1

BenzotBenzot-1

Cox models with time-varying coefficients to control for time-varying nature of covariates


When time-varying confounders are themselves affected by the previous treatment (SSRI use at t-1), conventional time-varying Cox model gives biased estimate

Adjusting-away part of treatment effect?

HFSSRItSSRIt-1

BenzotBenzot-1


In the presence of unmeasured common causes of confounders and outcome (U):

collider-stratification bias?

HFSSRItSSRIt-1

BenzotBenzot-1

U

Time - varying treatment

Adjusted for

Mondriaan BIFAP

HR 95% CI HR 95% CI

Crude 1.75 1.12, 2.72 2.09 1.89, 2.32

Gender 1.73 1.10, 2.69 2.07 1.87, 2.30

Gender +Age 2.36 1.51, 3.68 1.51 1.37, 1.68

Gender +Age + TCAt 2.59 1.63, 4.12 1.56 1.40, 1.73

Gender +Age + TCAt + Benzot 2.60 1.63, 4.16 1.54 1.38, 1.71

All Confounders* 2.62 1.63, 4.19 1.52 1.37, 1.69

Table 9. Associations Between SSRI Use and the Risk of Hip Fracture Using Time-Varying Cox Models


Table 10. Associations Between SSRI Use and the Risk of Hip Fracture Using Propensity Score Based Cox Analyses

Adjusted for

Mondriaan BIFAP

HR 95% CI HR 95% CI

Crude 1.75 1.12, 2.72 2.09 1.89, 1.71

PS* Stratification

Quintiles 2.64 1.63, 4.25 1.54 1.39, 1.71

Deciles 2.72 1.63, 4.54 1.53 1.38, 1.70

PS* Adjustment 2.82 1.63, 4.25 1.61 1.45, 1.78

Time-varying treatmentTable 11. Associations Between SSRI Use and the Risk of Hip Fracture

Using Propensity Score Based Cox Analyses

Adjusted forMondriaan BIFAP

HR 95% CI HR 95% CI

Without Accounting

for Censoring*

Crude 1.69 1.05, 2.67 2.14 1.91, 2.39

Gender 1.68 1.06, 2.67 2.11 1.89, 2.38

Gender +Age 2.46 1.55, 3.99 1.54 1.37, 1.72

Accounting for

Censoring**

Crude 1.73 1.08, 2.77 2.05 1.83, 2.30

Gender 1.71 1.07, 2.74 2.03 1.81, 2.28

Gender +Age 2.47 1.53, 3.98 1.51 1.35, 1.70

* Only inverse probability of treatment weights were used

** Combined inverse probability of treatment and censoring weights were used + Trimming at 1% and 99%


Differences between the various methods to adjust for time-dependent confounding (i.e., time-varying Cox and PS as well as inverse probability weighting of marginal structural models) were small.

The observed differences in treatment effects estimates between the datasets are likely attributable to different confounding information in the datasets.

Adequate information on (time-varying) confounding.

Conduct and Reporting PS Analysis: Literature review

M S Ali et al 2015

92 Articles were excluded:

63 Non clinical

20 Methodological

6 Other language

2 Systematic reviews

1 Editorials/letters

388 articles

PubMed search:

296 Articles available for review

Drug-related intervention: 108 (36.5%)

Clinical* 50 (16.9%)

Surgical intervention: 138 (46.6%)

Figure 1. Flow chart of abstracts or articles extraction for the systematic review

Reporting PS Analysis

PS matching is the commonly used approach.

Balance is often checked with PS matching followed by inverse probability weighting

P-values are the most commonly used tests.

Absolute standardized difference was used only in 25% of the studies in which balance was assessed.

M S Ali et al 2015

Reporting PS Analysis

Transparency in conducting the analysis

What needs to be reported??

Guidelines: STROBE or ENCePP

M S Ali et al 2015

PS Matching

- Matching algorithm, caliper width, and matching ratio used.

- Whether matching was with replacement. and matched nature of the data accounted in the analysis.

- Number of patients at the start and after matching.

- Number and characteristics of excluded patients (versus matched ones)

- The distribution of baseline characteristics between treated and control patients

in the matched and starting population.-Balance of Covariates between matched groups

♠

Variable Selection for PS Model

- Method used for variable selection - Whether empirical knowledge was considered - Whether variable’s association with treatment

and/or outcome

Propensity Score Estimation

- Method used to estimate the PS, e.g. logistic regression

- Variables and/or interaction terms included in the PS

Balance Assessment

- Balance measure used.- Quantifying the balance and whether imbalance on

covariates was detected after the final PS model.

Treatment Effect Estimation- Statistical method used.- Whether additional adjustment was made for

covariates.- Whether sensitivity analysis was performed.

Applying the PS methods

- Type of PS method employed.

Interpretation of Effect estimate

- The interpretation of the effect estimates in relation to the research question, target

- population & type of PS method used (ATE, ATT)

PS Weighting- The range (mean, max, min) of unstabilzed

and stabilized weights.- Variables included in the PS models for

both numerator and denominator of the weights.

- Whether weights were truncated and method used.

- Balance of Covariates in the weighted sample groups.

-

PS Stratification- The quantile used for stratification.- The overlap of PS with quantiles of PS using

plots or balance measures.- Balance of covariates with in quintiles of the

PS.

PS Adjustment

- Overlaps of the PS distribution between

treatment groups. *

- Whether linear relationship was checked

between outcome and the PS.

M Sanni Ali 2015

Limitations of PS

Unmeasured Confounding

Treatment Modification/Interaction

Multilevel Treatment Modeling

Dynamic Prescription Patterns/rare Exposure

Pooling Data on PS? Treatment guidelines

Alternatives: Disease Risk Score?

Acknowledgement

Olaf H. Klungel, PharmD, PhD1,2

Anthonius de Boer, MD, PhD1

Rolf HH Groenwold, MD, PhD1,2

Arno W. Hoes, MD, PhD2

Svetlana V. Belitser, MSc1

Kit C.B. Roes, PhD2

1 Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical

Sciences, Utrecht University, Utrecht.

2 Julius Centre for Health Science and Primary Care, University Medical Centre Utrecht, Utrecht.

Thank You!

Covariate Selection and Balance in Propensity Score Methods · Covariate Selection and Balance in...

Documents

Transcript of Covariate Selection and Balance in Propensity Score Methods · Covariate Selection and Balance in...