Multiplicity Correction in a Group-Sequential Oncology ... · PDF file• Largest...
44
Multiplicity Correction in a Group-Sequential Oncology Trial Including Subgroup Analyses and Multiple Primary Endpoints: Case Study European Statistical Meeting on Oncology and Survival Analysis Brussels, 17 November 2017 Agnes Balogh Bristol-Myers Squibb
Transcript of Multiplicity Correction in a Group-Sequential Oncology ... · PDF file• Largest...
Multiplicity Correction in a Group-Sequential Oncology Trial Including Subgroup Analyses and
Multiple Primary Endpoints Case Study
European Statistical Meeting on Oncology and Survival Analysis
Brussels 17 November 2017
Agnes Balogh
Bristol-Myers Squibb
Disclaimer
The case study presented here is an actual BMS study However
bull Study number disease setting drug names will not be provided
bull Numerical parameters of the statistical design are not the actual ones (rather hypothetical)
2
Outline
1 Study Design
2 Testing strategy
3 Concluding remarks
3
1 Study Design
4
Hypothetical Study Design
Control Experimental 1 Experimental 2
Population Metastatic Oncology
Randomize 111
Primary Population BM+ subpopulation
OS PFS OS PFS
Secondary Population All Randomized
OS PFS OS PFS
α = 001 α = 0015 α = 001 α = 0015
About 40 of the population prognostic and predictive
N = 1200
5
Group-Sequential Nature of the Design
First patient randomized
Enrolment
Time 1 Time 2
22months 29months 44months
PFS
OS (1)
PFS in BM+ is mature
OS in BM+ is mature
6
(1) Lan-DeMets α spending function used for the two BM+ OS endpoints OrsquoBrien and Fleming for the two All Randomized OS endpoints Pocock type of boundary
OS PFS Underlying Survival Curves
7
Legend
- - - - BM+ treatment
_______ All treatment
- - - - BM+ control
_______ All control
OS BM+ vs All Randomized PFS BM+ vs All Randomized
PFS
Rat
es
of Events in All 2 Arms vs in Control Arm Analysis will be conducted when
bull Time 1 PFS in BM+ is mature
bull Time 2 OS in BM+ is mature
Question should maturity be defined based on the of events in
1 All 3 arms in total OR
2 The control arm (requires independent group)
Statistical properties were investigated under 4 scenarios
8
Scenario Details
Protocol Assumptions
1 OS PFS Exp2 no treatment effect
OS and PFS no treatment effect in Experimental2 arm
2 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 30 in Experimental2 arm
3 Drop-out in all 3 arms 5 drop-out rate in all 3 arms
of Events in Total vs Control Simulation Results
Protocol assumptions
2 Exp2 better
1 Exp2 no treatment effect
3 5 drop-out in all 3 arms
9
Top line (dotted) is based on of events in control arm Bottom line (solid) is based on of events in all 3 arms
of Events in Total vs Control Statistical power bull For all 8 endpoints across all scenarios power is never lower if timing is later bull Largest difference for Exp1 in scenario1 (OS BM+ 11 OS All 13 PFS BM+ 5)
25 30 35 40 45 50 55 60 65 70
Months from First Patient Randomized
Time of Analysis (90 CI) under Different Scenarios
3 Testing Strategy
A Base Case
B Graphical Approach Introduction
C Chosen testing strategy bull Illustration of Sequentially Rejective Procedure
bull Power Gain
D Alternative testing strategies
10
11
Experimental1 vs Control Experimental2 vs Control
Testing Strategy Base Case
Time 1
0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002] 0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002]
Experimental1 vs Control Experimental2 vs Control
OS BM+
OS in All
[0009]
OS BM+
OS in All
[0009]
Time 2
For OS in BM+ information fraction assumed 70 (OrsquoBrien-Fleming)
Possibility for Improvement Graphical Approach
bull Historical overview bull Introduced in 2009 by Bretz et al (and Burman et al)
bull 2013 extension to group-sequential trials (Maurer Bretz)
bull 2017 Supported by the draft FDA guidance (Multiple Endpoints in Clinical Trials)
bull Closed test procedure =gt Strong control of family-wise error rate
bull A method to depict develop communicate a strategy consisting of Bonferroni-based sequential methods
Additional consideration PFS is not analysed at Time2 12
Experimental1 vs Control Experimental2 vs Control
Graphical Approach Introduction
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
Vertices = nodes
Endpoint-specific alpha asymp Local significance level
Edges = paths
Weight of edge
Blue expressions terminology of FDA guidance 13
Experimental1 vs Control Experimental2 vs Control
Improving Testing Strategy Add Edges
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
1
1
Along the green edges alpha can be passed only at Time1
14
15
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1a Time 1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
16
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1b Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Disclaimer
The case study presented here is an actual BMS study However
bull Study number disease setting drug names will not be provided
bull Numerical parameters of the statistical design are not the actual ones (rather hypothetical)
2
Outline
1 Study Design
2 Testing strategy
3 Concluding remarks
3
1 Study Design
4
Hypothetical Study Design
Control Experimental 1 Experimental 2
Population Metastatic Oncology
Randomize 111
Primary Population BM+ subpopulation
OS PFS OS PFS
Secondary Population All Randomized
OS PFS OS PFS
α = 001 α = 0015 α = 001 α = 0015
About 40 of the population prognostic and predictive
N = 1200
5
Group-Sequential Nature of the Design
First patient randomized
Enrolment
Time 1 Time 2
22months 29months 44months
PFS
OS (1)
PFS in BM+ is mature
OS in BM+ is mature
6
(1) Lan-DeMets α spending function used for the two BM+ OS endpoints OrsquoBrien and Fleming for the two All Randomized OS endpoints Pocock type of boundary
OS PFS Underlying Survival Curves
7
Legend
- - - - BM+ treatment
_______ All treatment
- - - - BM+ control
_______ All control
OS BM+ vs All Randomized PFS BM+ vs All Randomized
PFS
Rat
es
of Events in All 2 Arms vs in Control Arm Analysis will be conducted when
bull Time 1 PFS in BM+ is mature
bull Time 2 OS in BM+ is mature
Question should maturity be defined based on the of events in
1 All 3 arms in total OR
2 The control arm (requires independent group)
Statistical properties were investigated under 4 scenarios
8
Scenario Details
Protocol Assumptions
1 OS PFS Exp2 no treatment effect
OS and PFS no treatment effect in Experimental2 arm
2 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 30 in Experimental2 arm
3 Drop-out in all 3 arms 5 drop-out rate in all 3 arms
of Events in Total vs Control Simulation Results
Protocol assumptions
2 Exp2 better
1 Exp2 no treatment effect
3 5 drop-out in all 3 arms
9
Top line (dotted) is based on of events in control arm Bottom line (solid) is based on of events in all 3 arms
of Events in Total vs Control Statistical power bull For all 8 endpoints across all scenarios power is never lower if timing is later bull Largest difference for Exp1 in scenario1 (OS BM+ 11 OS All 13 PFS BM+ 5)
25 30 35 40 45 50 55 60 65 70
Months from First Patient Randomized
Time of Analysis (90 CI) under Different Scenarios
3 Testing Strategy
A Base Case
B Graphical Approach Introduction
C Chosen testing strategy bull Illustration of Sequentially Rejective Procedure
bull Power Gain
D Alternative testing strategies
10
11
Experimental1 vs Control Experimental2 vs Control
Testing Strategy Base Case
Time 1
0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002] 0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002]
Experimental1 vs Control Experimental2 vs Control
OS BM+
OS in All
[0009]
OS BM+
OS in All
[0009]
Time 2
For OS in BM+ information fraction assumed 70 (OrsquoBrien-Fleming)
Possibility for Improvement Graphical Approach
bull Historical overview bull Introduced in 2009 by Bretz et al (and Burman et al)
bull 2013 extension to group-sequential trials (Maurer Bretz)
bull 2017 Supported by the draft FDA guidance (Multiple Endpoints in Clinical Trials)
bull Closed test procedure =gt Strong control of family-wise error rate
bull A method to depict develop communicate a strategy consisting of Bonferroni-based sequential methods
Additional consideration PFS is not analysed at Time2 12
Experimental1 vs Control Experimental2 vs Control
Graphical Approach Introduction
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
Vertices = nodes
Endpoint-specific alpha asymp Local significance level
Edges = paths
Weight of edge
Blue expressions terminology of FDA guidance 13
Experimental1 vs Control Experimental2 vs Control
Improving Testing Strategy Add Edges
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
1
1
Along the green edges alpha can be passed only at Time1
14
15
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1a Time 1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
16
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1b Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Outline
1 Study Design
2 Testing strategy
3 Concluding remarks
3
1 Study Design
4
Hypothetical Study Design
Control Experimental 1 Experimental 2
Population Metastatic Oncology
Randomize 111
Primary Population BM+ subpopulation
OS PFS OS PFS
Secondary Population All Randomized
OS PFS OS PFS
α = 001 α = 0015 α = 001 α = 0015
About 40 of the population prognostic and predictive
N = 1200
5
Group-Sequential Nature of the Design
First patient randomized
Enrolment
Time 1 Time 2
22months 29months 44months
PFS
OS (1)
PFS in BM+ is mature
OS in BM+ is mature
6
(1) Lan-DeMets α spending function used for the two BM+ OS endpoints OrsquoBrien and Fleming for the two All Randomized OS endpoints Pocock type of boundary
OS PFS Underlying Survival Curves
7
Legend
- - - - BM+ treatment
_______ All treatment
- - - - BM+ control
_______ All control
OS BM+ vs All Randomized PFS BM+ vs All Randomized
PFS
Rat
es
of Events in All 2 Arms vs in Control Arm Analysis will be conducted when
bull Time 1 PFS in BM+ is mature
bull Time 2 OS in BM+ is mature
Question should maturity be defined based on the of events in
1 All 3 arms in total OR
2 The control arm (requires independent group)
Statistical properties were investigated under 4 scenarios
8
Scenario Details
Protocol Assumptions
1 OS PFS Exp2 no treatment effect
OS and PFS no treatment effect in Experimental2 arm
2 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 30 in Experimental2 arm
3 Drop-out in all 3 arms 5 drop-out rate in all 3 arms
of Events in Total vs Control Simulation Results
Protocol assumptions
2 Exp2 better
1 Exp2 no treatment effect
3 5 drop-out in all 3 arms
9
Top line (dotted) is based on of events in control arm Bottom line (solid) is based on of events in all 3 arms
of Events in Total vs Control Statistical power bull For all 8 endpoints across all scenarios power is never lower if timing is later bull Largest difference for Exp1 in scenario1 (OS BM+ 11 OS All 13 PFS BM+ 5)
25 30 35 40 45 50 55 60 65 70
Months from First Patient Randomized
Time of Analysis (90 CI) under Different Scenarios
3 Testing Strategy
A Base Case
B Graphical Approach Introduction
C Chosen testing strategy bull Illustration of Sequentially Rejective Procedure
bull Power Gain
D Alternative testing strategies
10
11
Experimental1 vs Control Experimental2 vs Control
Testing Strategy Base Case
Time 1
0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002] 0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002]
Experimental1 vs Control Experimental2 vs Control
OS BM+
OS in All
[0009]
OS BM+
OS in All
[0009]
Time 2
For OS in BM+ information fraction assumed 70 (OrsquoBrien-Fleming)
Possibility for Improvement Graphical Approach
bull Historical overview bull Introduced in 2009 by Bretz et al (and Burman et al)
bull 2013 extension to group-sequential trials (Maurer Bretz)
bull 2017 Supported by the draft FDA guidance (Multiple Endpoints in Clinical Trials)
bull Closed test procedure =gt Strong control of family-wise error rate
bull A method to depict develop communicate a strategy consisting of Bonferroni-based sequential methods
Additional consideration PFS is not analysed at Time2 12
Experimental1 vs Control Experimental2 vs Control
Graphical Approach Introduction
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
Vertices = nodes
Endpoint-specific alpha asymp Local significance level
Edges = paths
Weight of edge
Blue expressions terminology of FDA guidance 13
Experimental1 vs Control Experimental2 vs Control
Improving Testing Strategy Add Edges
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
1
1
Along the green edges alpha can be passed only at Time1
14
15
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1a Time 1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
16
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1b Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
1 Study Design
4
Hypothetical Study Design
Control Experimental 1 Experimental 2
Population Metastatic Oncology
Randomize 111
Primary Population BM+ subpopulation
OS PFS OS PFS
Secondary Population All Randomized
OS PFS OS PFS
α = 001 α = 0015 α = 001 α = 0015
About 40 of the population prognostic and predictive
N = 1200
5
Group-Sequential Nature of the Design
First patient randomized
Enrolment
Time 1 Time 2
22months 29months 44months
PFS
OS (1)
PFS in BM+ is mature
OS in BM+ is mature
6
(1) Lan-DeMets α spending function used for the two BM+ OS endpoints OrsquoBrien and Fleming for the two All Randomized OS endpoints Pocock type of boundary
OS PFS Underlying Survival Curves
7
Legend
- - - - BM+ treatment
_______ All treatment
- - - - BM+ control
_______ All control
OS BM+ vs All Randomized PFS BM+ vs All Randomized
PFS
Rat
es
of Events in All 2 Arms vs in Control Arm Analysis will be conducted when
bull Time 1 PFS in BM+ is mature
bull Time 2 OS in BM+ is mature
Question should maturity be defined based on the of events in
1 All 3 arms in total OR
2 The control arm (requires independent group)
Statistical properties were investigated under 4 scenarios
8
Scenario Details
Protocol Assumptions
1 OS PFS Exp2 no treatment effect
OS and PFS no treatment effect in Experimental2 arm
2 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 30 in Experimental2 arm
3 Drop-out in all 3 arms 5 drop-out rate in all 3 arms
of Events in Total vs Control Simulation Results
Protocol assumptions
2 Exp2 better
1 Exp2 no treatment effect
3 5 drop-out in all 3 arms
9
Top line (dotted) is based on of events in control arm Bottom line (solid) is based on of events in all 3 arms
of Events in Total vs Control Statistical power bull For all 8 endpoints across all scenarios power is never lower if timing is later bull Largest difference for Exp1 in scenario1 (OS BM+ 11 OS All 13 PFS BM+ 5)
25 30 35 40 45 50 55 60 65 70
Months from First Patient Randomized
Time of Analysis (90 CI) under Different Scenarios
3 Testing Strategy
A Base Case
B Graphical Approach Introduction
C Chosen testing strategy bull Illustration of Sequentially Rejective Procedure
bull Power Gain
D Alternative testing strategies
10
11
Experimental1 vs Control Experimental2 vs Control
Testing Strategy Base Case
Time 1
0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002] 0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002]
Experimental1 vs Control Experimental2 vs Control
OS BM+
OS in All
[0009]
OS BM+
OS in All
[0009]
Time 2
For OS in BM+ information fraction assumed 70 (OrsquoBrien-Fleming)
Possibility for Improvement Graphical Approach
bull Historical overview bull Introduced in 2009 by Bretz et al (and Burman et al)
bull 2013 extension to group-sequential trials (Maurer Bretz)
bull 2017 Supported by the draft FDA guidance (Multiple Endpoints in Clinical Trials)
bull Closed test procedure =gt Strong control of family-wise error rate
bull A method to depict develop communicate a strategy consisting of Bonferroni-based sequential methods
Additional consideration PFS is not analysed at Time2 12
Experimental1 vs Control Experimental2 vs Control
Graphical Approach Introduction
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
Vertices = nodes
Endpoint-specific alpha asymp Local significance level
Edges = paths
Weight of edge
Blue expressions terminology of FDA guidance 13
Experimental1 vs Control Experimental2 vs Control
Improving Testing Strategy Add Edges
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
1
1
Along the green edges alpha can be passed only at Time1
14
15
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1a Time 1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
16
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1b Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Hypothetical Study Design
Control Experimental 1 Experimental 2
Population Metastatic Oncology
Randomize 111
Primary Population BM+ subpopulation
OS PFS OS PFS
Secondary Population All Randomized
OS PFS OS PFS
α = 001 α = 0015 α = 001 α = 0015
About 40 of the population prognostic and predictive
N = 1200
5
Group-Sequential Nature of the Design
First patient randomized
Enrolment
Time 1 Time 2
22months 29months 44months
PFS
OS (1)
PFS in BM+ is mature
OS in BM+ is mature
6
(1) Lan-DeMets α spending function used for the two BM+ OS endpoints OrsquoBrien and Fleming for the two All Randomized OS endpoints Pocock type of boundary
OS PFS Underlying Survival Curves
7
Legend
- - - - BM+ treatment
_______ All treatment
- - - - BM+ control
_______ All control
OS BM+ vs All Randomized PFS BM+ vs All Randomized
PFS
Rat
es
of Events in All 2 Arms vs in Control Arm Analysis will be conducted when
bull Time 1 PFS in BM+ is mature
bull Time 2 OS in BM+ is mature
Question should maturity be defined based on the of events in
1 All 3 arms in total OR
2 The control arm (requires independent group)
Statistical properties were investigated under 4 scenarios
8
Scenario Details
Protocol Assumptions
1 OS PFS Exp2 no treatment effect
OS and PFS no treatment effect in Experimental2 arm
2 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 30 in Experimental2 arm
3 Drop-out in all 3 arms 5 drop-out rate in all 3 arms
of Events in Total vs Control Simulation Results
Protocol assumptions
2 Exp2 better
1 Exp2 no treatment effect
3 5 drop-out in all 3 arms
9
Top line (dotted) is based on of events in control arm Bottom line (solid) is based on of events in all 3 arms
of Events in Total vs Control Statistical power bull For all 8 endpoints across all scenarios power is never lower if timing is later bull Largest difference for Exp1 in scenario1 (OS BM+ 11 OS All 13 PFS BM+ 5)
25 30 35 40 45 50 55 60 65 70
Months from First Patient Randomized
Time of Analysis (90 CI) under Different Scenarios
3 Testing Strategy
A Base Case
B Graphical Approach Introduction
C Chosen testing strategy bull Illustration of Sequentially Rejective Procedure
bull Power Gain
D Alternative testing strategies
10
11
Experimental1 vs Control Experimental2 vs Control
Testing Strategy Base Case
Time 1
0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002] 0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002]
Experimental1 vs Control Experimental2 vs Control
OS BM+
OS in All
[0009]
OS BM+
OS in All
[0009]
Time 2
For OS in BM+ information fraction assumed 70 (OrsquoBrien-Fleming)
Possibility for Improvement Graphical Approach
bull Historical overview bull Introduced in 2009 by Bretz et al (and Burman et al)
bull 2013 extension to group-sequential trials (Maurer Bretz)
bull 2017 Supported by the draft FDA guidance (Multiple Endpoints in Clinical Trials)
bull Closed test procedure =gt Strong control of family-wise error rate
bull A method to depict develop communicate a strategy consisting of Bonferroni-based sequential methods
Additional consideration PFS is not analysed at Time2 12
Experimental1 vs Control Experimental2 vs Control
Graphical Approach Introduction
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
Vertices = nodes
Endpoint-specific alpha asymp Local significance level
Edges = paths
Weight of edge
Blue expressions terminology of FDA guidance 13
Experimental1 vs Control Experimental2 vs Control
Improving Testing Strategy Add Edges
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
1
1
Along the green edges alpha can be passed only at Time1
14
15
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1a Time 1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
16
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1b Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Group-Sequential Nature of the Design
First patient randomized
Enrolment
Time 1 Time 2
22months 29months 44months
PFS
OS (1)
PFS in BM+ is mature
OS in BM+ is mature
6
(1) Lan-DeMets α spending function used for the two BM+ OS endpoints OrsquoBrien and Fleming for the two All Randomized OS endpoints Pocock type of boundary
OS PFS Underlying Survival Curves
7
Legend
- - - - BM+ treatment
_______ All treatment
- - - - BM+ control
_______ All control
OS BM+ vs All Randomized PFS BM+ vs All Randomized
PFS
Rat
es
of Events in All 2 Arms vs in Control Arm Analysis will be conducted when
bull Time 1 PFS in BM+ is mature
bull Time 2 OS in BM+ is mature
Question should maturity be defined based on the of events in
1 All 3 arms in total OR
2 The control arm (requires independent group)
Statistical properties were investigated under 4 scenarios
8
Scenario Details
Protocol Assumptions
1 OS PFS Exp2 no treatment effect
OS and PFS no treatment effect in Experimental2 arm
2 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 30 in Experimental2 arm
3 Drop-out in all 3 arms 5 drop-out rate in all 3 arms
of Events in Total vs Control Simulation Results
Protocol assumptions
2 Exp2 better
1 Exp2 no treatment effect
3 5 drop-out in all 3 arms
9
Top line (dotted) is based on of events in control arm Bottom line (solid) is based on of events in all 3 arms
of Events in Total vs Control Statistical power bull For all 8 endpoints across all scenarios power is never lower if timing is later bull Largest difference for Exp1 in scenario1 (OS BM+ 11 OS All 13 PFS BM+ 5)
25 30 35 40 45 50 55 60 65 70
Months from First Patient Randomized
Time of Analysis (90 CI) under Different Scenarios
3 Testing Strategy
A Base Case
B Graphical Approach Introduction
C Chosen testing strategy bull Illustration of Sequentially Rejective Procedure
bull Power Gain
D Alternative testing strategies
10
11
Experimental1 vs Control Experimental2 vs Control
Testing Strategy Base Case
Time 1
0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002] 0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002]
Experimental1 vs Control Experimental2 vs Control
OS BM+
OS in All
[0009]
OS BM+
OS in All
[0009]
Time 2
For OS in BM+ information fraction assumed 70 (OrsquoBrien-Fleming)
Possibility for Improvement Graphical Approach
bull Historical overview bull Introduced in 2009 by Bretz et al (and Burman et al)
bull 2013 extension to group-sequential trials (Maurer Bretz)
bull 2017 Supported by the draft FDA guidance (Multiple Endpoints in Clinical Trials)
bull Closed test procedure =gt Strong control of family-wise error rate
bull A method to depict develop communicate a strategy consisting of Bonferroni-based sequential methods
Additional consideration PFS is not analysed at Time2 12
Experimental1 vs Control Experimental2 vs Control
Graphical Approach Introduction
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
Vertices = nodes
Endpoint-specific alpha asymp Local significance level
Edges = paths
Weight of edge
Blue expressions terminology of FDA guidance 13
Experimental1 vs Control Experimental2 vs Control
Improving Testing Strategy Add Edges
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
1
1
Along the green edges alpha can be passed only at Time1
14
15
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1a Time 1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
16
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1b Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
OS PFS Underlying Survival Curves
7
Legend
- - - - BM+ treatment
_______ All treatment
- - - - BM+ control
_______ All control
OS BM+ vs All Randomized PFS BM+ vs All Randomized
PFS
Rat
es
of Events in All 2 Arms vs in Control Arm Analysis will be conducted when
bull Time 1 PFS in BM+ is mature
bull Time 2 OS in BM+ is mature
Question should maturity be defined based on the of events in
1 All 3 arms in total OR
2 The control arm (requires independent group)
Statistical properties were investigated under 4 scenarios
8
Scenario Details
Protocol Assumptions
1 OS PFS Exp2 no treatment effect
OS and PFS no treatment effect in Experimental2 arm
2 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 30 in Experimental2 arm
3 Drop-out in all 3 arms 5 drop-out rate in all 3 arms
of Events in Total vs Control Simulation Results
Protocol assumptions
2 Exp2 better
1 Exp2 no treatment effect
3 5 drop-out in all 3 arms
9
Top line (dotted) is based on of events in control arm Bottom line (solid) is based on of events in all 3 arms
of Events in Total vs Control Statistical power bull For all 8 endpoints across all scenarios power is never lower if timing is later bull Largest difference for Exp1 in scenario1 (OS BM+ 11 OS All 13 PFS BM+ 5)
25 30 35 40 45 50 55 60 65 70
Months from First Patient Randomized
Time of Analysis (90 CI) under Different Scenarios
3 Testing Strategy
A Base Case
B Graphical Approach Introduction
C Chosen testing strategy bull Illustration of Sequentially Rejective Procedure
bull Power Gain
D Alternative testing strategies
10
11
Experimental1 vs Control Experimental2 vs Control
Testing Strategy Base Case
Time 1
0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002] 0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002]
Experimental1 vs Control Experimental2 vs Control
OS BM+
OS in All
[0009]
OS BM+
OS in All
[0009]
Time 2
For OS in BM+ information fraction assumed 70 (OrsquoBrien-Fleming)
Possibility for Improvement Graphical Approach
bull Historical overview bull Introduced in 2009 by Bretz et al (and Burman et al)
bull 2013 extension to group-sequential trials (Maurer Bretz)
bull 2017 Supported by the draft FDA guidance (Multiple Endpoints in Clinical Trials)
bull Closed test procedure =gt Strong control of family-wise error rate
bull A method to depict develop communicate a strategy consisting of Bonferroni-based sequential methods
Additional consideration PFS is not analysed at Time2 12
Experimental1 vs Control Experimental2 vs Control
Graphical Approach Introduction
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
Vertices = nodes
Endpoint-specific alpha asymp Local significance level
Edges = paths
Weight of edge
Blue expressions terminology of FDA guidance 13
Experimental1 vs Control Experimental2 vs Control
Improving Testing Strategy Add Edges
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
1
1
Along the green edges alpha can be passed only at Time1
14
15
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1a Time 1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
16
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1b Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
of Events in All 2 Arms vs in Control Arm Analysis will be conducted when
bull Time 1 PFS in BM+ is mature
bull Time 2 OS in BM+ is mature
Question should maturity be defined based on the of events in
1 All 3 arms in total OR
2 The control arm (requires independent group)
Statistical properties were investigated under 4 scenarios
8
Scenario Details
Protocol Assumptions
1 OS PFS Exp2 no treatment effect
OS and PFS no treatment effect in Experimental2 arm
2 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 30 in Experimental2 arm
3 Drop-out in all 3 arms 5 drop-out rate in all 3 arms
of Events in Total vs Control Simulation Results
Protocol assumptions
2 Exp2 better
1 Exp2 no treatment effect
3 5 drop-out in all 3 arms
9
Top line (dotted) is based on of events in control arm Bottom line (solid) is based on of events in all 3 arms
of Events in Total vs Control Statistical power bull For all 8 endpoints across all scenarios power is never lower if timing is later bull Largest difference for Exp1 in scenario1 (OS BM+ 11 OS All 13 PFS BM+ 5)
25 30 35 40 45 50 55 60 65 70
Months from First Patient Randomized
Time of Analysis (90 CI) under Different Scenarios
3 Testing Strategy
A Base Case
B Graphical Approach Introduction
C Chosen testing strategy bull Illustration of Sequentially Rejective Procedure
bull Power Gain
D Alternative testing strategies
10
11
Experimental1 vs Control Experimental2 vs Control
Testing Strategy Base Case
Time 1
0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002] 0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002]
Experimental1 vs Control Experimental2 vs Control
OS BM+
OS in All
[0009]
OS BM+
OS in All
[0009]
Time 2
For OS in BM+ information fraction assumed 70 (OrsquoBrien-Fleming)
Possibility for Improvement Graphical Approach
bull Historical overview bull Introduced in 2009 by Bretz et al (and Burman et al)
bull 2013 extension to group-sequential trials (Maurer Bretz)
bull 2017 Supported by the draft FDA guidance (Multiple Endpoints in Clinical Trials)
bull Closed test procedure =gt Strong control of family-wise error rate
bull A method to depict develop communicate a strategy consisting of Bonferroni-based sequential methods
Additional consideration PFS is not analysed at Time2 12
Experimental1 vs Control Experimental2 vs Control
Graphical Approach Introduction
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
Vertices = nodes
Endpoint-specific alpha asymp Local significance level
Edges = paths
Weight of edge
Blue expressions terminology of FDA guidance 13
Experimental1 vs Control Experimental2 vs Control
Improving Testing Strategy Add Edges
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
1
1
Along the green edges alpha can be passed only at Time1
14
15
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1a Time 1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
16
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1b Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
of Events in Total vs Control Simulation Results
Protocol assumptions
2 Exp2 better
1 Exp2 no treatment effect
3 5 drop-out in all 3 arms
9
Top line (dotted) is based on of events in control arm Bottom line (solid) is based on of events in all 3 arms
of Events in Total vs Control Statistical power bull For all 8 endpoints across all scenarios power is never lower if timing is later bull Largest difference for Exp1 in scenario1 (OS BM+ 11 OS All 13 PFS BM+ 5)
25 30 35 40 45 50 55 60 65 70
Months from First Patient Randomized
Time of Analysis (90 CI) under Different Scenarios
3 Testing Strategy
A Base Case
B Graphical Approach Introduction
C Chosen testing strategy bull Illustration of Sequentially Rejective Procedure
bull Power Gain
D Alternative testing strategies
10
11
Experimental1 vs Control Experimental2 vs Control
Testing Strategy Base Case
Time 1
0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002] 0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002]
Experimental1 vs Control Experimental2 vs Control
OS BM+
OS in All
[0009]
OS BM+
OS in All
[0009]
Time 2
For OS in BM+ information fraction assumed 70 (OrsquoBrien-Fleming)
Possibility for Improvement Graphical Approach
bull Historical overview bull Introduced in 2009 by Bretz et al (and Burman et al)
bull 2013 extension to group-sequential trials (Maurer Bretz)
bull 2017 Supported by the draft FDA guidance (Multiple Endpoints in Clinical Trials)
bull Closed test procedure =gt Strong control of family-wise error rate
bull A method to depict develop communicate a strategy consisting of Bonferroni-based sequential methods
Additional consideration PFS is not analysed at Time2 12
Experimental1 vs Control Experimental2 vs Control
Graphical Approach Introduction
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
Vertices = nodes
Endpoint-specific alpha asymp Local significance level
Edges = paths
Weight of edge
Blue expressions terminology of FDA guidance 13
Experimental1 vs Control Experimental2 vs Control
Improving Testing Strategy Add Edges
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
1
1
Along the green edges alpha can be passed only at Time1
14
15
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1a Time 1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
16
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1b Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
3 Testing Strategy
A Base Case
B Graphical Approach Introduction
C Chosen testing strategy bull Illustration of Sequentially Rejective Procedure
bull Power Gain
D Alternative testing strategies
10
11
Experimental1 vs Control Experimental2 vs Control
Testing Strategy Base Case
Time 1
0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002] 0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002]
Experimental1 vs Control Experimental2 vs Control
OS BM+
OS in All
[0009]
OS BM+
OS in All
[0009]
Time 2
For OS in BM+ information fraction assumed 70 (OrsquoBrien-Fleming)
Possibility for Improvement Graphical Approach
bull Historical overview bull Introduced in 2009 by Bretz et al (and Burman et al)
bull 2013 extension to group-sequential trials (Maurer Bretz)
bull 2017 Supported by the draft FDA guidance (Multiple Endpoints in Clinical Trials)
bull Closed test procedure =gt Strong control of family-wise error rate
bull A method to depict develop communicate a strategy consisting of Bonferroni-based sequential methods
Additional consideration PFS is not analysed at Time2 12
Experimental1 vs Control Experimental2 vs Control
Graphical Approach Introduction
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
Vertices = nodes
Endpoint-specific alpha asymp Local significance level
Edges = paths
Weight of edge
Blue expressions terminology of FDA guidance 13
Experimental1 vs Control Experimental2 vs Control
Improving Testing Strategy Add Edges
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
1
1
Along the green edges alpha can be passed only at Time1
14
15
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1a Time 1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
16
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1b Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
11
Experimental1 vs Control Experimental2 vs Control
Testing Strategy Base Case
Time 1
0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002] 0015
PFS BM+
PFS in All
OS BM+
OS in All
[0002]
Experimental1 vs Control Experimental2 vs Control
OS BM+
OS in All
[0009]
OS BM+
OS in All
[0009]
Time 2
For OS in BM+ information fraction assumed 70 (OrsquoBrien-Fleming)
Possibility for Improvement Graphical Approach
bull Historical overview bull Introduced in 2009 by Bretz et al (and Burman et al)
bull 2013 extension to group-sequential trials (Maurer Bretz)
bull 2017 Supported by the draft FDA guidance (Multiple Endpoints in Clinical Trials)
bull Closed test procedure =gt Strong control of family-wise error rate
bull A method to depict develop communicate a strategy consisting of Bonferroni-based sequential methods
Additional consideration PFS is not analysed at Time2 12
Experimental1 vs Control Experimental2 vs Control
Graphical Approach Introduction
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
Vertices = nodes
Endpoint-specific alpha asymp Local significance level
Edges = paths
Weight of edge
Blue expressions terminology of FDA guidance 13
Experimental1 vs Control Experimental2 vs Control
Improving Testing Strategy Add Edges
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
1
1
Along the green edges alpha can be passed only at Time1
14
15
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1a Time 1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
16
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1b Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Possibility for Improvement Graphical Approach
bull Historical overview bull Introduced in 2009 by Bretz et al (and Burman et al)
bull 2013 extension to group-sequential trials (Maurer Bretz)
bull 2017 Supported by the draft FDA guidance (Multiple Endpoints in Clinical Trials)
bull Closed test procedure =gt Strong control of family-wise error rate
bull A method to depict develop communicate a strategy consisting of Bonferroni-based sequential methods
Additional consideration PFS is not analysed at Time2 12
Experimental1 vs Control Experimental2 vs Control
Graphical Approach Introduction
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
Vertices = nodes
Endpoint-specific alpha asymp Local significance level
Edges = paths
Weight of edge
Blue expressions terminology of FDA guidance 13
Experimental1 vs Control Experimental2 vs Control
Improving Testing Strategy Add Edges
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
1
1
Along the green edges alpha can be passed only at Time1
14
15
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1a Time 1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
16
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1b Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Experimental1 vs Control Experimental2 vs Control
Graphical Approach Introduction
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
Vertices = nodes
Endpoint-specific alpha asymp Local significance level
Edges = paths
Weight of edge
Blue expressions terminology of FDA guidance 13
Experimental1 vs Control Experimental2 vs Control
Improving Testing Strategy Add Edges
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
1
1
Along the green edges alpha can be passed only at Time1
14
15
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1a Time 1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
16
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1b Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Experimental1 vs Control Experimental2 vs Control
Improving Testing Strategy Add Edges
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
0015
0
1
PFS BM+
PFS in All
OS BM+
OS in All
1
001
0
1
1
1
Along the green edges alpha can be passed only at Time1
14
15
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1a Time 1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
16
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1b Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
15
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1a Time 1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
16
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1b Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
16
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 1b Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
17
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2a Time 1
0015
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
18
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 2b Time 1
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
1
Along the green edges alpha can be passed only at Time1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
19
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3a Time 2
OS BM+
OS In All
1
0025 [00060023]
0
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
20
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3b Time 2
OS BM+
OS In All
1
0025 [00060023]
0
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
21
Experimental1 vs Control Experimental2 vs Control
Illustration of Sequentially Rejective Procedure Step 3c Time 2
OS In All
0025 [002001]
PFS BM+
PFS In All
OS BM+
OS In All
1
001 [00020009]
0
1
1
For OS information fraction assumed BM+ 70 (OrsquoBrien-Fleming) AC 75 (Pocock)
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
1 1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
1
1
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +171 00 +179 00
All +176 00 +179 00
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ +66 +57
All +77 +73 22
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
How Does the Power Gain Change under Different Assumptions
Endpoint Modified
Scenario Details
OS 1 OS larger difference in treatment effect by BM status
HR (for non-cure patients and after delay) increased by 10 in the BM- and decreased by 10 in the BM+ population (while keeping HR for all randomized the same) (a)
2 OS shorter delay 10 shorter delay in all 4 OS endpoints (a)
OS PFS 3 OS PFS Exp2 better OS and PFS treatment effect (delay cure rate HR in non-cure patients) improved by 10 in Experimental2 arm
4 OS PFS Exp2 worse OS and PFS treatment effect (delay cure rate HR in non-cure patients) worsened by 10 in Experimental2 arm
5 OS PFS longer delay OS delay increased with 50 PFS delay with 300
PFS 6 PFS no delay No delay in any of the 4 PFS endpoints
Time of analysis is generated based on of events in all 3 arms 23
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
0
01
02
03
04
05
06
07
0
01
02
03
04
05
06
07
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time1
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC
05
055
06
065
07
075
08
085
09
095
1
05
055
06
065
07
075
08
085
09
095
1
PROTOCOL 1 OS Larger BM effect 2 OS shorter delay 3 Exp2 better 4 Exp2 worse 5 OS PFS longer delay 6 PFS no delay
Benefit of Additional Edges in Statistical Power of OS Time2
OS1+
OS1All
OS2+
OS2All
OS1
OS1AC
OS2
OS2AC24
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Experimental1 vs Control Experimental2 vs Control
Alternative1 Secure All Endpoints within Comparison
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
ε
1 - ε 1 - ε
25
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dashed edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative1 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
ε
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
ε
1-ε 1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 170 + 14 + 178 + 17
All + 176 + 16 + 178 + 20
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 64 + 56
All + 75 + 71 26
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Experimental1 vs Control Experimental2 vs Control
Alternative2 Giving BM+ Maximum Chance
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
001
0
1
ε2 ε2
1 - ε 1 - ε
1
1 1
ε2 ε2
1
27
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative2 Change in Power vs Base Case (under Protocol Assumptions)
1 - ε 1 - ε
1
1 1
ε2
0015
0
PFS BM+
PFS In All
OS BM
+
OS In All
0002
0
1
0015
0
PFS BM+
PFS In All
OS BM+
OS In All
0002
0
1
ε2 ε2 ε2
1
Time 1 Time 2
OS BM+
OS In All
0002
0
OS BM+
OS In All
0002
0
05 05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 135 + 08 + 138 + 08
All - 197 - 80 - 19 - 79
+20
+15
+10
+5
0
-5
-10
-15
-20
28
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 37 + 32
All - 67 - 68
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
4 Concluding Remarks
29
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Concluding Remarks MTP
bull Limitation of the results on statistical power bull OS and PFS were generated independently
bull Identifying the testing strategy for an actual clinical trial always requires the input of several functions and the power results represent only one aspect of these considerations
bull Using graphical approach may result in overpowered endpoints However this still may be desirable reasons bull Protocol assumptions are complicated =gt just one detail off can result in power drop
(even in the presence of meaningful treatment effect)
bull At an interim analysis probability of crossing the efficacy boundary increases =gt gives larger chance for earlier read-out
30
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Thank you for your attention
31
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
References
bull Bretz F Maurer W Brannath W Posch M A graphical approach to sequentially rejective multiple test procedures Statistics in Medicine 2009 28 586-604
bull Burman C-F Sonesson C and Guilbaud O (2009) A recycling framework for the construction of Bonferroni-based multiple tests Statistics in Medicine 28 739ndash761
bull Bretz F Posch M Glimm E Klinglmueller F Maurer W Rohmeyer K Graphical approaches for multiple comparison procedures using weighted Bonferroni Simes or parametric tests Biometrical Journal 2011 53(6) 894-913
bull Ye Y Li A Liu L and Yao B (2013) A group sequential Holm procedure with multiple primary endpoints Statistics in Medicine 32 1112-1124
bull Maurer W and Bretz F (2013a) ldquoGraphical Multiple Test Procedures With Memoryrdquo Statistics in Medicine DOI 101002sim5711
bull Willi MAURER and Frank BRETZ Multiple Testing in Group Sequential Trials Using Graphical Approaches American Statistical Association Statistics in Biopharmaceutical Research November 2013 Vol5 No4
bull Xi D Tamhane AC Allocating recycled significance levels in group sequential procedures for multiple endpoints Biom J 2015 Jan57(1)90-107
bull Tamhane A C Mehta C R and Liu L (2010) Testing a primary and a secondary endpoint in a group sequential design Biometrics 66 1174ndash1184
32
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Back-up
33
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Basic Parameters of the Study Design Table 1 Basic Parameters of the Study Design
Overall Survival Progression-Free Survival
BM+ BM- All Rand BM+ BM- All Rand
randomized Total (per comparison)
asymp 480 (asymp 320) (c)
asymp 720 (asymp 480) (c)
1200 (800) (fixed)
As OS As OS As OS
Significance level (a) 001 0015
Delayed Effect (b) 4m 5m 1m 1m
Cure Rate in Experimental (b) 15 10 0 0
HR after Delay Effect (b) (d) 06 075 056 075
Overall HR (b) Time1 Time2 070 061 084 074 077 069 063 084 072
Median controlexperimental 8m 12m 9m 12m 9m 12m 4m 64m 4m 47m 4m 55m
events Total Time1 Time2 asymp 285 385 (fixed)
asymp 693 asymp 963 380 (fixed) asymp 989
events per comparison Time1 Time2
asymp 198 asymp 266 asymp 478 asymp 661 asymp 261 asymp 672
(a) Initially allocated endpoint-specific alpha used for sample size calculation (b) For each comparison with the two experimental arms (c) Based on an expected BM+ proportion of 40 (Total number randomized is fixed) (d) As assumed for non-cure patients
34
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Literature Overview
Graphical approach Bonferroni-based (Bretz et al 2009)
Non-parametric extensions (Bretz et al 2011)
GSP extension special case (Ye et al 2013)
GSP general allows for selection of recycling stages (Xi Tamhane 2015)
GSP extension general (Maurer Bretz 2013)
Graphical approach Bonferroni-based (Burman 2009)
GSP = group-sequential procedure Dark blue boxes referenced by FDA guidance
Graphs with memory (Maurer Bretz 2013a)
-649
-648 -577 -459
35
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Multiplicity History of Regulatory Guidances bull E9 Statistical Principles for Clinical Trials (1998 Section 56)
bull Pre-specification of MTP bull EMA Points to Consider on Multiplicity Issues in Clinical Trials (2002)
bull Strong control of FWER bull EMA Concept paper on the need for a guideline on multiplicity issues in
clinical trials (Draft 2012) =gt Nov2012 Workshop bull EMA Guideline on multiplicity issues in clinical trials bull FDA Guidance document on multiplicity issues in clinical trials bull PMDA no multiplicity guidance (ldquoMTPs do not necessarily match with
labellingrdquo)
36
DRAFT Released for public
consultation in 2017
Related document within BMS Reporting Observed Significance Levels (p-values) in Clinical Study Reports Regulatory Documents and Publications 2014 (Stephane Munier)
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Multiplicity Latest Regulatory Guidancersquos (2017) EMA Guideline on multiplicity issues
in clinical trials FDA Guidance document on
multiplicity issues in clinical trials
Principles bull Family-wise strong control bull Pre-specification of multiplicity adjustment
Secondary Endpoints bull Secondary endpoints expressing supportive evidence needs no MTP
bull Secondary endpoints that may become base for additional claims need MTP
(Section 6)
Distinction between different types of secondary endpoints is not clear all require MTP (Section IIIA)
Scope Similar (not the same)
Subgroups Section 7 (also own guidance) Excluded (another guidance)
Estimation and descriptive statistics Section 10 Not included
Graphical approach Not mentioned Explicitly encouraged
37
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Experimental1 vs Control Experimental2 vs Control
Alternative4 Importance of Experimental1 PFS
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
0015
0
1
PFS BM+
PFS In All
OS BM+
OS In All
1
001
0
1
ε
1
1 - ε
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain 38
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Time 1 Time 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative4 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
0015
0
1
PFS BM
+
PFS In All
OS BM
+
OS In All
1
0002
0
1
ε
1
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
1
0009
0
OS BM
+
OS In All
1
0002
0
ε
1
1-ε
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 166 + 13 + 131 0
AC + 162 + 19 + 116 0
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 56 + 30
AC + 69 + 40 39
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Experimental1 vs Control Experimental2 vs Control
Alternative3 Importance of both OS BM+
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
0015
0
05
PFS BM+
PFS In All
OS BM+
OS In All
05
001
0
05 05
Along the green edges alpha can be passed only at IA Along the blue edge recycled alpha can be used only at interim Along the dotted edges alpha is passed only if for the given vertex
only dotted outgoing edges remain
05
05
40
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Timepoint 1 Timepoint 2
Experimental1 vs Control Experimental2 vs Control Experimental1 vs Control Experimental2 vs Control
Alternative3 Change in Power vs Base Case (under Protocol Assumptions)
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
05
0002
0
05
0015
0
PFS BM
+
PFS In All
OS BM
+
OS In All
0002
0
For OS information fraction assumed BM+ 70 AC 75
+20
+15
+10
+5
0
-5
-10
-15
-20
OS BM
+
OS In All
0009
0
OS BM
+
OS In All
0002
0
05
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 104 00 + 105 00
All + 87 - 14 + 84 - 16
Exp1 vs Control Exp2 vs Control
OS PFS OS PFS
BM+ + 46 NA + 40 NA
All + 49 NA + 42 NA
05 05 05 05 05
41
05 05 05 05
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
B) Final OS Look Adjusting Significance Levels bull The time of the final OS analysis will be determined by the number of
BM+ OS events in the 3 arms or in the control arm
bull As a result the number of events for each OS comparison is a random variable by design
bull Consequence bull Time 1 Assume none of the OS endpoints was rejected bull Time 2 If the of events for a given comparison is more (less) than expected
=gt the information fraction and nominal significance level used at Time 1 was larger (smaller) than what we should have used
For the final look critical values for OS will be re-calculated using the number of exactly observed events at the final analysis
The effect of adjustment is limited (always lt 0004 and in general lt 0001)
42
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Illustration Effect of Final Alpha Adjustment
43 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Legend Observed IF at Time 1 lt Q1 Q1 - Q2 Q2 - Q3 Q3 lt
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
Illustration Effect of Final Alpha Adjustment
44 The first 1000 samples generated under protocol assumptions number of events fixed in control arm assumed a local significance level of 005
