An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn...
Transcript of An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn...
![Page 1: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/1.jpg)
1
1
An Interim Analysis Example
Mark A. Weaver, PhDFamily Health International
Office of AIDS Research, NIHICSSC, FHI
Goa, India, September 2009
![Page 2: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/2.jpg)
2
2
Example• We are designing an HIV vaccine trial
– HIV negative, high-risk women will be randomized to either vaccine or placebo
• We desire 90% power to detect a 50% reduction in HIV incidence (HR = 0.50), 2-sided α = 0.05– Tomorrow, we will see that power depends on
number of events (HIV infections)– The events provide all of the “information” for test – For now, just know that trial requires at least 88
events
![Page 3: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/3.jpg)
3
3
Analysis Plan
• Primary efficacy analysis:H0: HR = 1 vs. HA: HR ≠ 1
• Plan to test this using Cox PH modelh(t) = h0(t) exp{ β * Trt }
where Trt = 1 for vaccine, 0 for placeboβ = log(HR) (β < 0 for HR < 1, β > 0 for HR > 1)
• Test statistic:)ˆ( SE
ˆ
ββ
=Z
![Page 4: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/4.jpg)
4
4
Plan for Early Stopping
• Desire to stop the trial early in the event of:1. clear early evidence of efficacy2. clear early evidence of harm (wrt HIV)3. early evidence of harm based on additional
safety data (AEs, labs, vital signs, etc.)
• Also might decide to stop for futility – What is “futility”?
• Could also stop for poor trial quality, slow enrollment, high loss, etc.
![Page 5: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/5.jpg)
5
5
“No single statistical test or monitoring procedure ought to be used as a strict rule for decision-making, but rather as one piece of
evidence to be integrated with other evidence.”
Friedman, Furberg, and DeMetsFundamentals of Clinical Trials, 3rd ed.
![Page 6: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/6.jpg)
6
6
Planning the Interim Analyses
• Plan to test primary hypothesis at 4 points, 3 interim analyses and 1 final analysis– After 22, 44, 66, and 88 events
• Goal: maintain the overall probability of a type 1 error (i.e., α) at specified 0.05 level
• Choose a “group sequential” stopping boundary to control α1. Haybittle-Peto2. Pocock3. O’Brien-Fleming
![Page 7: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/7.jpg)
7
7
-5
-4
-3
-2
-1
0
1
2
3
4
5
22 44 66 88
Haybittle-PetoO'Brien-FlemingPocock
Z Continue
Reject H0, harm
Reject H0, efficacy
Do Not Reject H0
Events
![Page 8: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/8.jpg)
8
8
-5
-4
-3
-2
-1
0
1
2
3
4
5
22 44 66 88
Haybittle-PetoO'Brien-FlemingPocock
Z ContinueDo Not Reject H0
Events
Reject H0
Reject H0
![Page 9: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/9.jpg)
9
9
-5
-4
-3
-2
-1
0
1
2
3
4
5
22 44 66 88
Haybittle-PetoO'Brien-FlemingPocock
Z ContinueDo Not Reject H0
Events
Reject H0
Reject H0
![Page 10: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/10.jpg)
10
10
-5
-4
-3
-2
-1
0
1
2
3
4
5
22 44 66 88
Haybittle-PetoO'Brien-FlemingPocock
Z Continue
Reject H0
Reject H0
Do Not Reject H0
Events
![Page 11: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/11.jpg)
11
11
Monitoring the Trial
• Suppose the trial started enrolling and following patients on January 1, 2008.
• On June 15, 2008, event number 22 detected• Data were locked, interim analysis run, DSMB
meeting held following week• 6 events in vaccine arm, 16 in placebo arm
– Estimated HR = 0.38– β = -0.97, SE(β) = 0.48– Z = -0.97 / 0.48 = -2.02 → unadjusted p = 0.0434– Looking good, but can we stop at this point?
![Page 12: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/12.jpg)
12
12
-5
-4
-3
-2
-1
0
1
2
3
4
5
22 44 66 88
Z
Events
Z = -4.05
![Page 13: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/13.jpg)
13
13
Monitoring the Trial
• On January 13, 2009, event number 44 detected• Data were locked, interim analysis run, DSMB
meeting held following week• 14 events in vaccine arm, 30 in placebo arm
– Estimated HR = 0.47– β = -0.76, SE(β) = 0.32– Z = -0.76 / 0.32 = -2.38 → unadjusted p = 0.0174– Wow, that’s really “significant” … but can we stop at this
point?
![Page 14: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/14.jpg)
14
14
-5
-4
-3
-2
-1
0
1
2
3
4
5
22 44 66 88
Z
Events
Z = -2.86
![Page 15: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/15.jpg)
15
15
Monitoring the Trial
• On May 31, 2009, event number 66 detected• Data were locked, interim analysis run, DSMB
meeting held following week• 21 events in vaccine arm, 45 in placebo arm
– Estimated HR = 0.47 (same as before)– β = -0.76, SE(β) = 0.26– Z = -0.76 / 0.26 = -2.92 → unadjusted p = 0.0036– Wow! … can we stop now?
![Page 16: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/16.jpg)
16
16
-5
-4
-3
-2
-1
0
1
2
3
4
5
22 44 66 88
Z
Events
Z = -2.34
![Page 17: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/17.jpg)
17
17
SUCCESS!!
But, what else should a DSMB consider prior to recommending that the trial stop?
![Page 18: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/18.jpg)
18
18
Monitoring the Trial
• DSMB noted increased incidence of troubling AE (severe apparent allergic reactions) in vaccine arm– Due to a bad batch, possibly?
• DSMB recommends to not stop the trial– Note, this is all that the project team would get to know!
• Trial proceeded to completion (88 events)• 35 events in vaccine arm, 53 in placebo arm
– Estimated HR =0.66– β = -0.42, SE(β) = 0.21– Z = -0.42 / 0.21 = -2.00 → unadjusted p = 0.0456
![Page 19: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/19.jpg)
19
19
-5
-4
-3
-2
-1
0
1
2
3
4
5
22 44 66 88
Z
Events
Z = -2.02
![Page 20: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/20.jpg)
20
20
What if All Doesn’t Go According to Plan?
• Boundaries technically designed for fixed number of equally spaced analyses– What if 77 events (rather than 66) had occurred prior
to the 3rd look at the data?– Or, what if DSMB had requested an additional look
between 66 and 88 events to explore AE issue?– O’Brien-Fleming and Pocock boundaries are fairly
robust to unequally spaced analyses, but neither accommodates additional, unplanned analyses
• What about Haybittle-Peto procedure?
![Page 21: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/21.jpg)
21
21
Flexible Monitoring – “Alpha Spending”• Lan and DeMets developed a very flexible
boundary procedure that accommodates unequal timing, additional looks, even extending the trial– Called “alpha spending” functions– Researchers can choose to “spend” their alpha (i.e.,
conduct interim analyses) any way they want– Ensures that total alpha “spent” is no more than 0.05
(or whatever was specified)
• There are other spending function options
![Page 22: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/22.jpg)
22
22
Alpha Spending Functions
Events
α
0.00
0.01
0.02
0.03
0.04
0.05
0 22 44 66 88
O'Brien-FlemingPocock
![Page 23: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/23.jpg)
23
23
Flexible Monitoring – “Alpha Spending”
• In protocol, we typically specify: “The Lan-DeMets spending function with O’Brien-Fleming type boundaries will be employed to preserve the overall one-sided type I error rate for effectiveness at the 0.025 level, regardless of the timing of the analysis.”
• Software available for calculating (East, “ldbounds” package in R – this is a “free”download at http://www.r-project.org/, others?)
![Page 24: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/24.jpg)
24
24
Alpha Spending - Example• Suppose our 3rd interim analysis happened after
77 rather than 66 events (i.e, after 88% of total rather than 75%)– L-D spending function would have used Z = -2.14
rather than -2.34 at 77 events– At 88 events, would use z = -2.02
• Suppose DSMB requested additional interim analysis at 77 events given safety issues?– L-D spending function at 77 events would have used
Z = -2.21 (having used -2.34 at 66 events)– At 88 events, would use z = -2.06 (slight penalty)
![Page 25: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/25.jpg)
25
25
What if our first look had gone other way?
-5
-4
-3
-2
-1
0
1
2
3
4
5
22 44 66 88
Events
Zunadjusted p ≈ 0.001
Reject H0, harm
![Page 26: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/26.jpg)
26
26
Asymmetric Boundaries
• It can be unethical to continue a study to get definitive evidence that product is harmful
• Typically need to be more “liberal” wrt harm• “Trending” toward harm might be enough for
most DSMB members, anyway• Also might want to stop for futility
– E.g., if it’s highly improbable that study will provide a significantly favorable result given current data
– Fiscally logical to kill an ineffective product ASAP and move onto the next one
![Page 27: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/27.jpg)
27
27
“Potential Harm” Boundary
Continue
Reject H0, efficacy
Stop for “potential harm”
Do Not Reject H0
Events
Z
-5
-4
-3
-2
-1
0
1
2
3
4
5
22 44 66 88
![Page 28: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/28.jpg)
28
28
Futility Boundary
-5
-4
-3
-2
-1
0
1
2
3
4
5
22 44 66 88
Events
Z
Continue
Reject H0, efficacy
Stop for futility, do not reject H0
![Page 29: An Interim Analysis Example - ICSSCicssc.org/Documents/AdvBiosGoa/Tab 25.00_InterimAnalysis.pdfAn Interim Analysis Example Mark A. Weaver, PhD ... – O’Brien-Fleming and Pocock](https://reader031.fdocuments.net/reader031/viewer/2022021418/5aa0125c7f8b9a67178dbd73/html5/thumbnails/29.jpg)
29
29
Questions?