David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015...
-
Upload
kimberly-teresa-hunter -
Category
Documents
-
view
214 -
download
1
description
Transcript of David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015...
![Page 1: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/1.jpg)
David L. DeMets Lectures
Exploratory Analyses:Why Do We Need Particular Caution?
November 12, 2015
Thomas R. Fleming, Ph.D.Professor, Dept. of Biostatistics
University of Washington
* Fleming TR “Clinical Trials: Discerning Hype from Substance”• Annals of Internal Medicine 2010; 153:400-406
![Page 2: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/2.jpg)
![Page 3: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/3.jpg)
David DeMets is a visionary figure in Biostatistics and Biomedical Informatics… a scholar, researcher, teacher, and leader, evidenced by his founding and chairing a leading department of biostatistics and medical informatics and by his developing seminal methods for clinical trials …his papers on interim analysis and roles of DMCs, are landmarks in defining biostatistical features of randomized clinical trials. His impressive leadership has been widely praised by many academic institutions & professional organizations.
David L. DeMets: Institute of Medicine Citation
![Page 4: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/4.jpg)
SEMINAL CONTRIBUTION OF TOM FLEMING & DAVE DEMETS: PRINCIPLES & PRACTICAL SUGGESTIONS FOR
DATA MONITORING COMMITTEES
![Page 5: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/5.jpg)
SEMINAL CONTRIBUTION OF TOM FLEMING & DAVE DEMETS: PRINCIPLES & PRACTICAL SUGGESTIONS FOR
DATA MONITORING COMMITTEES
![Page 6: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/6.jpg)
Data Driven Hypothesis for the Cancer Risk with Vytorin in Aortic-Valve Stenosis
SEAS Trial N CA. Incidence CA. Deaths Vytorin 944 101 37 Placebo 929 65 20 Relative Risk: 1.55 1.78
95% C.I.: (1.13, 2.12) (1.03. 3.11)
IMPROVE-IT & SHARP Trials N CA. Incidence CA. Deaths Vytorin 10,391 313 97 Control 10,298 326 72 Relative Risk: 0.96 1.34 95% C.I.: (0.82, 1.12) (0.98, 1.84)
![Page 7: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/7.jpg)
CPCRA #007: Study Design
Patient Population
ddIGroup
ddCGroup
Unblinded
ZDVddI
active
ZDVddC
active
ZDVddC
placebo
ZDVddI
placebo
Blinded
600 600
400 200 200 400
![Page 8: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/8.jpg)
Data Driven Hypothesis for the Relative Effectof ddI Placebo vs. ddC Placebo in HIV/AIDS
• CPCRA #007 11/93 5/95 ZDV ZDV ZDV ZDV ddI ddC ddI ddC Placebo Placebo Placebo Placebo
n 172 168 188 187Prog/Death 42 28 100 95 Death 17 2 75 66 All Events 73 37 210 202
![Page 9: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/9.jpg)
Interest in “Positive” Results in Clinical Trials Industry Sponsors
~ Company profits, ↑ value of stock options, promotion Government Sponsors ~ Claims of success in advancing health care ~ Leverage for ↑ in federal funding Journal Editors (Publication bias) Academic Investigators / Caregivers ~ Increased ability to publish results
↑ professional stature, earlier promotion, ↑ salary ~ Desire to offer more therapeutic options to patients….Result: Wide Spread & Significant Conflicts of Interest
![Page 10: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/10.jpg)
~ What is the definition of a successful clinical trial?
A very common response: “A clinical trial that achieves a positive result”
Bias for “Positive” Results in Clinical Trials
![Page 11: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/11.jpg)
~ What is the definition of a successful clinical trial?
A very common response: “A clinical trial that achieves a positive result”
The proper scientific response: “A clinical trial that reliably answers the
questions the trial was designed to address”
Bias for “Positive” Results in Clinical Trials
![Page 12: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/12.jpg)
Confirmatory vs. Exploratory Analyses
• Hyp. Confirmation vs. Hyp. Generation ~ Post-hoc analyses & Random High Bias (new endpoints, new analyses, interim analyses subgroup analyses, covariate adjustments)
![Page 13: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/13.jpg)
Confirmatory vs. Exploratory Analyses
• Clinical Endpoints in Pulmonary Arterial Hypertension ~ Overall survival ~ Quality of Life: SF-36 (8 domains), Borg Dyspnea Score
• ~ NYHA Functional Class ~ 6MWT: @18 wk, 24 wk, 48 wk, etc. ~ Time to Clinical Worsening
Death, PAH Hosp, L.T., (NYHA↑ & 6MWT↓)
• Analysis Methods ~ Normally distributed: T-test, ANOVA, Wilcoxon ~ Time to event: Log-rank, Cox Regression ~ Dichotomous: Fisher’s Exact Test, Pearson χ2
![Page 14: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/14.jpg)
Confirmatory vs. Exploratory Analyses
• Biomarker Endpoints (Hemodynamic parameters) ~ Pulmonary Arterial Pressure
~ Systolic & Diastolic Systemic Arterial Pressure
~ Systemic & Pulmonary Vascular Resistance
~ Heart Rate & Cardiac Output
• Analyses over Calendar Time • ~ Normally distributed: T-test, ANOVA, Wilcoxon • ~ Time to event: Log-rank, Cox Regression
~ Dichotomous: Fisher’s Exact Test, Pearson χ2
![Page 15: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/15.jpg)
Confirmatory vs. Exploratory Analyses
• Subgroup Analysis & Prognostic Covariate Adjustment
~ WHO PAH Functional Class: I v II v III v IV
~ Etiology: Idiopathic PAH, Assoc w CTD, SLE, Other
~ Baseline Walking Distance: < 325 v > 325 meters ~ Gender: male v female
~ Age: By decade
~ Ethnicity: White v Black v Asian v Other
~ mean PAP: < 50 v > 50
Epoprostenol +/─ Sildenafil
![Page 16: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/16.jpg)
Confirmatory vs. Exploratory Analyses
• Hyp. Confirmation vs. Hyp. Generation ~ Post-hoc analyses & Random High Bias (new endpoints, new analyses, interim analyses subgroup analyses, covariate adjustments)
Illustrations and Motivation:
![Page 17: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/17.jpg)
Confirmatory vs. Exploratory Analyses
• Hyp. Confirmation vs. Hyp. Generation ~ Post-hoc analyses & Random High Bias (new endpoints, new analyses, interim analyses subgroup analyses, covariate adjustments)
Illustrations and Motivation: Maternity Wards, Baseball & Clinical Research
20 vs 2: (.71, .99), 2p = 0.0001
![Page 18: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/18.jpg)
An Illustration of Exploratory Analyses:
Post-hoc Subgroup Analyses
Surgical Adjuvant Therapyof Colorectal Cancer
5-FU + LevamisoleLevamisoleControl
R
![Page 19: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/19.jpg)
Surgical Adjuvant Therapy: Colorectal CancerS
urv i
v ing
, %
0 1 2 3 4 5 6
100 -
80 -
60 -
40 -
20 -
0
Years from randomization
NCCTG Trial
5-FU+LEV n=81LEV n=85Control n=81
![Page 20: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/20.jpg)
NORTH CENTRAL TREATMENT GROUP STUDYLooking at Treatment Effect on Overall Survival
0 2 4 6 8 10 12 14
100 -
80 -
60 -
40 -
20 -
0
Females OnlyAt Risk Death 5-Yr Estimate 44 21 57% 43 29 40%
Males OnlyAt Risk Death 5-Yr Estimate 37 25 51% 38 24 47%
0 2 4 6 8 10 12
Years from Registration
5-FU+LevamisoleFollow-Up Only
-
-
-
-
-
![Page 21: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/21.jpg)
Surgical Adjuvant Therapy: Colorectal CancerS
urv i
v ing
, %
0 1 2 3 4 5 6
100 -
80 -
60 -
40 -
20 -
0
Years from randomization
NCCTG Trial
5-FU+LEV n=81LEV n=85Control n=81
![Page 22: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/22.jpg)
Surgical Adjuvant Therapy: Colorectal CancerS
urv i
v ing
, %
0 1 2 3 4 5 6
100 -
80 -
60 -
40 -
20 -
0
Years from randomization
NCCTG Trial Cancer Intergroup Trial
0 1 2 3 4 5 6 7 8 9
100 -
80 -
60 -
40 -
20 -
0
Years from randomization
5-FU+LEV n=81LEV n=85Control n=81
5-FU+LEV n=304LEV n=310Control n=315
![Page 23: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/23.jpg)
INTERGROUP STUDY 0035Looking at Treatment Effect on Overall Survival
100 -
80 -
60 -
40 -
20 -
0
Females OnlyAt Risk Death 5-Yr Estimate 163 74 58% 149 77 54%
Males OnlyAt Risk Death 5-Yr Estimate 141 47 70% 166 91 51%
0 2 4 6 8 10 12
Years from Registration
5-FU+LevamisoleFollow-Up Only
-
-
-
-
-
0 2 4 6 8 10 12
![Page 24: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/24.jpg)
Duke’s C Colon Cancer Adjuvant
Percent ↓ in Death Rate: 5-FU + Levamisole Control
Analysis North Central Intergroup Group Treatment Study Group Study # 0035 (n = 162) (n = 619)
All patients 28% 33%
Female 43% 15%Male 9% 50%
Young 40% 23%Old 13% 41%
![Page 25: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/25.jpg)
An Illustration of Exploratory Analyses:
Post-hoc Subgroup Analyses
Radiation Treatment in Rectal CancerPrincess Margaret Hospital
Pre-operative R.T.
ControlR
![Page 26: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/26.jpg)
Years
100
90
80
70
60
50
40
30
20
101 2 3 4 5 6 7
Survival %
Control
Irradiated
PMH--Toronto Study# = no. at risk
Survival of Patients with Rectal Carcinomain Control and Irradiated Groups
60
65
![Page 27: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/27.jpg)
Years
100
90
80
70
60
50
40
30
20
101 2 3 4 5 6 7
Survival %
Control
Irradiated
PMH--Toronto Study# = no. at risk
Survival of Patients with Dukes’ Stage C RectalCarcinoma in Control and Irradiated Groups
22
16
2p = 0.01
![Page 28: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/28.jpg)
Survival by Treatment Allocated
100
80
60
40
20
0
Time, mo
Surv
iva l
rate
, %
0 6 12 18 24 30 36 42 48 54 60 66
No XRT (275)Single fraction (277)Multiple fractions (272)
Med. Research Council Study P = 0.92
![Page 29: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/29.jpg)
Survival by Treatment for Dukes’ C Cases
100
80
60
40
20
0
Time, mo
Surv
iva l
rate
, %
0 6 12 18 24 30 36 42 48 54 60 66
No XRT (111)Single fraction (110)Multiple fractions (79)
Med. Research Council Study
![Page 30: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/30.jpg)
Confirmatory vs. Exploratory Analyses
• Hyp. Confirmation vs. Hyp. Generation ~ Post-hoc analyses & Random High Bias (new endpoints, new analyses, interim analyses subgroup analyses, covariate adjustments)
Illustrations and Motivation: Maternity Wards, Baseball & Clinical Research
![Page 31: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/31.jpg)
Years
100
90
80
70
60
50
40
30
20
101 2 3 4 5 6 7
Survival %
Control
Irradiated
PMH--Toronto Study# = no. at risk
Survival of Patients with Dukes’ Stage C RectalCarcinoma in Control and Irradiated Groups
22
16
![Page 32: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/32.jpg)
Survival by Treatment for Dukes’ C Cases
100
80
60
40
20
0
Time, mo
Surv
iva l
rate
, %
0 6 12 18 24 30 36 42 48 54 60 66
No XRT (111)Single fraction (110)Multiple fractions (79)
Med. Research Council Study
![Page 33: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/33.jpg)
Thrombolytics inAcute Myocardial Infarction
• GISSI (Lancet ’86) - SK reduces mortality by 20%
![Page 34: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/34.jpg)
Thrombolytics inAcute Myocardial Infarction
• GISSI (Lancet ’86) - SK reduces mortality by 20%
confined to: anterior MI < 65 years < 6 hours from symptom onset
![Page 35: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/35.jpg)
Thrombolytics inAcute Myocardial Infarction
• GISSI (Lancet ’86) - SK reduces mortality by 20%
confined to: anterior MI < 65 years < 6 hours from symptom onset
- Subset restriction not confirmed by ISIS-2, ASSET, AIMS
![Page 36: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/36.jpg)
Thrombolytics inAcute Myocardial Infarction
• GISSI (Lancet ’86) - SK reduces mortality by 20%
confined to: anterior MI < 65 years < 6 hours from symptom onset
- Subset restriction not confirmed by ISIS-2, ASSET, AIMS - While in ISIS-2:
Aspirin beneficial overall…
![Page 37: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/37.jpg)
Thrombolytics inAcute Myocardial Infarction
• GISSI (Lancet ’86) - SK reduces mortality by 20%
confined to: anterior MI < 65 years < 6 hours from symptom onset
- Subset restriction not confirmed by ISIS-2, ASSET, AIMS - While in ISIS-2:
Aspirin beneficial overall… … yet harmful to patients with
astrological signs Libra and Gemini
![Page 38: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/38.jpg)
Can Efficacy or Safety Signals Discovered in Exploratory Analyses Be Viewed to be Reliable Results?
• Criteria to be simultaneously satisfied:
< < P-values (e.g., Tysabri & PML)
Biologically plausible effect
White Paper Illustration
Confirmed by external results
![Page 39: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/39.jpg)
Years
100
90
80
70
60
50
40
30
20
101 2 3 4 5 6 7
Survival %
Control
Irradiated
PMH--Toronto Study# = no. at risk
Survival of Patients with Dukes’ Stage C RectalCarcinoma in Control and Irradiated Groups
22
16
![Page 40: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/40.jpg)
Survival by Treatment for Dukes’ C Cases
100
80
60
40
20
0
Time, mo
Surv
iva l
rate
, %
0 6 12 18 24 30 36 42 48 54 60 66
No XRT (111)Single fraction (110)Multiple fractions (79)
Med. Research Council Study
![Page 41: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/41.jpg)
Surgical Adjuvant Therapy: Colorectal CancerS
urv i
v ing
, %
0 1 2 3 4 5 6
100 -
80 -
60 -
40 -
20 -
0
Years from randomization
NCCTG Trial
5-FU+LEV n=81LEV n=85Control n=81
![Page 42: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/42.jpg)
Surgical Adjuvant TherapyOf Colorectal Cancer
Surv
ivin
g, %
0 1 2 3 4 5 6
100 -
80 -
60 -
40 -
20 -
0
Years from randomization
NCCTG Trial Cancer Intergroup Trial
0 1 2 3 4 5 6 7 8 9
100 -
80 -
60 -
40 -
20 -
0
Years from randomization
5-FU+LEV n=91Levamisole n=85Control n=86
5-FU+LEV n=304Levamisole n=310Control n=315
![Page 43: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/43.jpg)
Of all experimental interventions studied in colon cancer,suppose only 4% are truly positive & 96% are truly negative. Suppose the “false negative error rate” is = 0.10 (so the “statistical power” is 1- = 0.90 ) & Suppose the “false positive error rate” is = 0.025 Then, the probability a trial positive will be a true positive is 36 / 60 = 0.60
RESULT OF TRUTHEXPERIMENT Positive Negative
Positive 36 24 60Negative 4 936 940
40 960 1000
![Page 44: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/44.jpg)
Of all experimental interventions studied, suppose 60% are truly positive & 40% are truly negative Suppose the “false negative error rate” is = 0.10 (so the “statistical power” is 1- = 0.90 ) & Suppose the “false positive error rate” is = 0.025 Then, the probability a trial positive will be a true positive is 540 / 550 = 0.98
RESULT OF TRUTHEXPERIMENT Positive Negative
Positive 540 10 550Negative 60 390 450
600 400 1000
![Page 45: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/45.jpg)
Surgical Adjuvant Therapy Of Colorectal Cancer
Sur
v iv i
ng, %
0 1 2 3 4 5 6
100 -
80 -
60 -
40 -
20 -
0
Years from randomization
NCCTG Trial Cancer Intergroup Trial
0 1 2 3 4 5 6 7 8 9
100 -
80 -
60 -
40 -
20 -
0
Years from randomization
5-FU+LEV n=91LEV n=85Control n=86
5-FU+LEV n=304LEV n=310Control n=315
![Page 46: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/46.jpg)
“It isn’t so much the things we don’t knowthat get us in trouble.
It’s the things we know that aren’t so”.
—Artemus Ward (1834-1867)
![Page 47: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/47.jpg)
Some Conclusions
• P-values are only interpretable when you understandthe sampling context from which they were
derived
• Random High bias is real
• Exploratory Analyses usually should be viewedto be “Hypothesis Generating”
• Confirmatory Trialsgreatly enhance the reliability of conclusions
![Page 48: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/48.jpg)
Confirmatory vs. Exploratory Analyses
• Hyp. Confirmation vs. Hyp. Generation ~ Post-hoc analyses & Regression to Mean (new endpoints, new analyses, interim analyses subgroup analyses, covariate adjustments)
Illustrations and Motivation: Maternity Wards, Baseball & Clinical Research
20 vs 2: (.71, .99), 2p = 0.0001 Meta-Analysis: 31 vs 13: (.55, .83), 2p = 0.0096
![Page 49: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/49.jpg)
Bias for “Positive” Results in Clinical Trials
Protocol Specified Primary Objective of the Clinical trial:
• Very frequent wording: ~ “ To establish that the experimental regimen is safe and effective”
![Page 50: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/50.jpg)
Bias for “Positive” Results in Clinical Trials
Protocol Specified Primary Objective of the Clinical trial:
• Very frequent wording: ~ “ To establish that the experimental regimen is safe and effective”
• Scientifically unbiased wording: ~ “ To evaluate whether the experimental regimen is safe and effective”
…building a story with supportive analyses…
![Page 51: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/51.jpg)
Bias for “Positive” Results in Clinical Trials
…Andrew Fleming’s insight from Psychology…
“Cognitive Dissonance”
…The Harvard Professor’s Course…
…The Apparent Lack of Benefit in Males…
![Page 52: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/52.jpg)
Interest in “Positive” Results in Clinical Trials
• Abetimus Sodium: Reducing Renal Flare Rate in Lupus
• Trial #1: Time to renal flare: Minimal effect, (2p = 0.51)
![Page 53: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/53.jpg)
Interest in “Positive” Results in Clinical Trials
• Abetimus Sodium: Reducing Renal Flare Rate in Lupus
• Trial #1: Time to renal flare: Minimal effect, (2p = 0.51)…exploratory high affinity subgroup: 2p = 0.007
• Trial #2 conducted in high affinity subgroup: Time to renal flare:
![Page 54: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/54.jpg)
Interest in “Positive” Results in Clinical Trials
• Abetimus Sodium: Reducing Renal Flare Rate in Lupus
• Trial #1: Time to renal flare: Minimal effect, (2p = 0.51)…exploratory high affinity subgroup: 2p = 0.007
• Trial #2 conducted in high affinity subgroup: Time to renal flare: Minimal non-significant effect
![Page 55: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/55.jpg)
Interest in “Positive” Results in Clinical Trials
• Abetimus Sodium: Reducing Renal Flare Rate in Lupus
• Trial #1: Time to renal flare: Minimal effect, (2p = 0.51)…exploratory high affinity subgroup: 2p = 0.007
• Trial #2 conducted in high affinity subgroup: Time to renal flare: Minimal non-significant effect
…exploratory truncation at 12 months is favorable
![Page 56: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/56.jpg)
Interest in “Positive” Results in Clinical Trials
• Abetimus Sodium: Reducing Renal Flare Rate in Lupus
• Trial #1: Time to renal flare: Minimal effect, (2p = 0.51)…exploratory high affinity subgroup: 2p = 0.007
• Trial #2 conducted in high affinity subgroup: Time to renal flare: Minimal non-significant effect
…exploratory truncation at 12 months is favorable
• Trial #3 conducted in high affinity subgroupwith prespecified truncation at 12 months follow-up:
![Page 57: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/57.jpg)
Interest in “Positive” Results in Clinical Trials
• Abetimus Sodium: Reducing Renal Flare Rate in Lupus
• Trial #1: Time to renal flare: Minimal effect, (2p = 0.51)…exploratory high affinity subgroup: 2p = 0.007
• Trial #2 conducted in high affinity subgroup: Time to renal flare: Minimal non-significant effect
…exploratory truncation at 12 months is favorable
• Trial #3 conducted in high affinity subgroupwith prespecified truncation at 12 months follow-up: …early termination by DMC for futility.
![Page 58: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/58.jpg)
“If you Torture Data Long Enough,They will Confess
![Page 59: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/59.jpg)
“The Goal of Clinical Research:
Principles & Insights
![Page 60: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/60.jpg)
“The Goal of Clinical Research:To Determine Whether,
Not to Establish,the Experimental Regimen
Is Safe and Effective”
Principles & Insights
Fleming TR “Clinical Trials: Discerning Hype from Substance”Annals of Internal Medicine 2010; 153:400-406
![Page 61: David L. DeMets Lectures Exploratory Analyses: Why Do We Need Particular Caution? November 12, 2015 Thomas R. Fleming, Ph.D. Professor, Dept. of Biostatistics.](https://reader035.fdocuments.net/reader035/viewer/2022070605/5a4d1af47f8b9ab059980708/html5/thumbnails/61.jpg)