1

1

Improving

Reliability, Transparency, and Reproducibility

of database research without transmitting patient-level databases

Sebastian Schneeweiss, MD, ScD Professor of Medicine and Epidemiology

Division of Pharmacoepidemiology and Pharmacoeconomics, Dept of Medicine, Brigham & Women’s Hospital/ Harvard Medical School

Potential conflicts of interest

v PI, Harvard-Brigham & Women’s Hospital Drug Safety Research Center (FDA)

v Chair, Methods Core of the FDA Sentinel System v Member, national PCORI Methods Committee v Consulting in past year:

§  WHISCON LLC, Aetion Inc. (incl. equity)

v Investigator-initiated research grants to the Brigham & Women’s Hospital from Novartis, Genentech, Boehringer Ingelheim

v Grants/contracts from NIH, AHRQ, PCORI, FDA, IMI

2

Reproducibility in the life sciences

3

How can we make drug safety database studies more trusted?

4

v 1) Reduce bias in Healthcare database analyses: §  Reduce confounding §  Reduce time-related biases §  Reduce measurement-related biases

v 2) Reduce investigator error §  Have user interface and state-of-the-art workflows that

ensure valid and transparent choices re. design & analysis

v 3) Make studies reproducible w/o sharing data §  Have complete reporting enabling 100% reproducibility §  Share analytic environments not data

A.  Surveillance-related B.  Misclassification C.  Misspecification

A.  Temporality B.  Immortal time C.  Time on market

5

2) Reduce Investigator Error

6

2

Reliable Causal Analyses

7 4

Intrinsic Study Characteristics q  Internal validity (bias) q  External validity (generalizability, transportability) q  Precision q  Heterogeneity in risk or benefit (personalized evidence) q  Ethical consideration (equipoise) External Study Characteristics q  Timeliness (rapidly changing technology, policy needs) q  Logistical constraints (study size, complexity, cost) q  Data availability, quality, completeness

From the PCORI Methods Committee report

9

Subgroup Analysis ?

Basic Design Consideration

Subgroup definition

Prior pharmacology knowledge

Prior clinical Knowledge

Yes

Cohort study (case-control, case-cohort sampling)

Exposure/outcome considerations

Exposure definition Outcome Definition

Comparison group considerations Clinical meaningfulness

Incident user design considerations

Exposure risk window considerations Case validation necessary?

Specificity and sensitivity of measurement

Yes Consider case-crossover design

no

Meaningful exposure variation within patients?

Schneeweiss 2010

Basic Study Design for safety studies Define best practice workflows

Example:  A  typical  Drug  Safety  Study  Workflow  

10  

11

UIs that guide the investigator along a problem-based workflow

Selecting data source and patients in transparent and reproducible ways

Deciding on risk adjustment Deciding on follow-up plan

Deciding on comparison group

CONFIDENTIAL 12

Define the analytic approach (ITT vs. AT), covariate identification period, follow-up time period, censoring etc....

3

CONFIDENTIAL 13

Specify the outcome model for the primary and secondary outcomes, propensity score matching, trimming, stratifying approaches...

14

Intrinsic Study Characteristics q  Internal validity (bias) q  External validity (generalizability, transportability) q  Precision q  Heterogeneity in risk or benefit (personalized evidence) q  Ethical consideration (equipoise) External Study Characteristics q  Timeliness (rapidly changing technology, policy needs) q  Logistical constraints (study size, complexity, cost) q  Data availability, quality, completeness

From the PCORI Methods Committee report

Avoiding obviously wrong choices will reduce heterogeneity of results

15

Risk Difference

0

X X X

Inve

stig

ator

1

Limited heterogeneity from valid design and analysis choices Extreme and unnecessary heterogeneity from invalid choices (eg some of OMOP’s choices)*

* Susan Gruber, 2014 OMOP Symposium Inve

stig

ator

2

Inve

stig

ator

3

3) Make Studies Reproducible

16

17 !

!

! Peng et al. 2006

Reproducibility in Epidemiology Reproducibility in Epidemiology (not Replicability)

18 Peng et al. 2006

Can we do this with healthcare databases?

We should do this already!

We should do this already! Can we do this with healthcare databases?

4

Shared cloud-based analytics

19

Solution for investigators that cannot store data on their cloud:

20

Every  analysis  generates  a  comprehensive  and  readable  report  that  allows  100%  reproducCon  of  the  research  …  

21  

Methods p3-13

Results p14-30

Appendix p31-65

…  by  providing  all  details  regarding  coding  and  methods  

22  

Good News

v Reliability of database research can be improved through structured approaches

v Reproducibility can be achieved if §  We completely and precisely record all choices made

during design and analysis §  We share analytic code (R, SAS, etc.) §  We share data

v Sharing the analytic environment gets around the inability to freely share most healthcare databases

23