Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints...
Transcript of Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints...
![Page 1: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/1.jpg)
Analysing Survival Endpoints in Randomized Clinical
Trials using Generalized Pairwise Comparisons
Dr Julien PERON October 2016
Department of Biostatistics HCL – LBBE UCBL Department of Medical oncology HCL – LBBE UCBL
![Page 2: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/2.jpg)
Outline
• The standard procedure of generalized pairwise comparisons
• An extension for right-censored data
• A patient-oriented measure of treatment benefit
• Benefit-risk balance • NCIC PA.3 trial => erlotinib in pancreatic cancer
2
![Page 3: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/3.jpg)
Outline
• The standard procedure of generalized pairwise comparisons
• NCIC PA.3 trial => erlotinib in pancreatic cancer
3
![Page 4: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/4.jpg)
Methods – Pairwise comparisons
Let xi be the outcome of
i th subject in T (i = 1. … . n )
R
Control (C ) Treatment (T )
Let yj be the outcome of
j th subject in C (j = 1. … . m )
Yj Xi
favors T (favorable)
favors C (unfavorable)
pairwise comparison
Neutral or Uninformative 4
Buyse M. stat in med 2010
![Page 5: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/5.jpg)
Methods – Definition of thresholds
Coutinuous outcome
5 Buyse. stat in med 2010
Pair Rating
𝑥𝑖 𝑦𝑗 > Favorable
𝑥𝑖 𝑦𝑗 < ( ) Unfavorable
𝑥𝑖 𝑦𝑗 ≤ Neutral
𝑥𝑖 or 𝑦𝑗 missing Uninformative
![Page 6: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/6.jpg)
Methods – Definition of priority
First priority outcome
Second priority outcome
Pair rating
Favorable NA Favorable
Unfavorable NA Unfavorable
Neutral/Uninf Favorable Favorable
Neutral/Uninf Unfavorable Unfavorable
Neutral/Uninf Neutral/Uninf Neutral/Uninf
6 Buyse. stat in med 2010
![Page 7: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/7.jpg)
Methods – Definition of priority
First priority outcome
Second priority outcome
Pair rating
Favorable NA Favorable
Unfavorable NA Unfavorable
Neutral/Uninf Favorable Favorable
Neutral/Uninf Unfavorable Unfavorable
Neutral/Uninf Neutral/Uninf Neutral/Uninf
7 Buyse. stat in med 2010
![Page 8: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/8.jpg)
Methods – Definition of priority
First priority outcome
Second priority outcome
Pair rating
Favorable NA Favorable
Unfavorable NA Unfavorable
Neutral/Uninf Favorable Favorable
Neutral/Uninf Unfavorable Unfavorable
Neutral/Uninf Neutral/Uninf Neutral/Uninf
8 Buyse. stat in med 2010
![Page 9: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/9.jpg)
Methods – Standard procedure for pairwise scoring
Δ in named «net chance of a better outcome»
9 Buyse. stat in med 2010
n
i
m
j
Uijnm
U1 1
1
otherwise 0
eunfavorabl is pair when the1
favorable is pair when the1
Y j,X i
Y j,X i
U ij
![Page 10: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/10.jpg)
• 𝑥𝑖0 and 𝑦𝑗
0 : time-to-event
• 𝑥𝑖 and 𝑦𝑗 : time-to-observation
• Event indicator :
𝛿𝑖 = 1 𝑖𝑓 𝑥𝑖 = 𝑥𝑖
0
0 𝑖𝑓 𝑥𝑖 < 𝑥𝑖0 in group T
휀𝑗 = 1 𝑖𝑓 𝑦𝑗 = 𝑦𝑗
0
0 𝑖𝑓 𝑦𝑗 < 𝑦𝑗0 in group C
• Survival function:
𝑆𝑇𝑡𝑡 𝑡 = ℙ 𝑥𝑖0 ≥ 𝑡 and 𝑆𝐶𝑡𝑟𝑙 𝑡 = ℙ 𝑦𝑗
0 ≥ 𝑡
10
Some notations
![Page 11: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/11.jpg)
11
(𝛿𝑖 , 휀𝑗) 𝑥𝑖 − 𝑦𝑗 ≥ 𝜏 𝑥𝑖 − 𝑦𝑗 ≤ −𝜏 𝑥𝑖 − 𝑦𝑗 < 𝜏
(1, 1) Favorable Unfavorable Neutral
(0, 1) Favorable Uninformative Uninformative
(1, 0) Uninformative Unfavorable Uninformative
(0, 0) Uninformative Uninformative Uninformative
Buyse M. Stat in med, 2010
The standard procedure to include time-to-event’ outcome
![Page 12: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/12.jpg)
12
0,5
1,0
Surv
ival
Pro
bab
ility
0,0
Time
Patient i : censoring
Treatment group
Control group
Patient j : event
The standard procedure to include time-to-event outcome
Gehan. Biometrika, 1965
![Page 13: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/13.jpg)
Outline
• An extension for right-censored data
• NCIC PA.3 trial => erlotinib in pancreatic cancer
13
![Page 14: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/14.jpg)
Based on the Kaplan-Meier estimate of the survival function
𝕡 ( 𝑥𝑖0> 𝑦𝑗
0) 𝑥𝑖0> 𝑥𝑖 =
𝑆 𝑇𝑡𝑡 𝑦𝑗
𝑆 𝑇𝑡𝑡(𝑥𝑖)=
0,5
0,8
The extended procedure taking into account ‘non-informative’ pairs
0,5
1,0
Surv
ival
Pro
bab
ility
0,0
Time
0,8
Péron J et al, SMMR 2016
Patient i : censoring
Patient j : event
Treatment group
Control group
![Page 15: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/15.jpg)
15
𝕡 ( 𝑥𝑖0> 𝑦𝑗
0) 𝑥𝑖0> 𝑥𝑖 , 𝑦𝑗
0> 𝑦𝑗 = − 𝑆 𝑇𝑡𝑡 𝑡
𝑆 𝑇𝑡𝑡 𝑥𝑖 𝑆 𝐶𝑡𝑟𝑙 𝑦𝑗∙ 𝑆 𝐶𝑡𝑟𝑙 𝑡
+ − 𝑆 𝐶𝑡𝑟𝑙 𝑡−
∞
𝑡>𝑦𝑗
Efron, Berkeley Symp, 1967
0,5
1,0
0,0
When the estimation of the survival function is discontinue :
Surv
ival
Pro
bab
ility
Time
Patient i : censoring
Treatment group
Control group
Patient j : censoring
The extended procedure taking into account ‘non-informative’ pairs
![Page 16: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/16.jpg)
For pairs that can not be decidely classified because of censoring, we compute:
ℙ 𝑥𝑖0 > 𝑦𝑗
0 + 𝜏 , et ℙ 𝑦𝑗0 > 𝑥𝑖
0 + 𝜏 , et ℙ 𝑥𝑖0 − 𝑦𝑗
0 < 𝜏
The pairwise score is: 𝑠𝑖𝑗 = ℙ 𝑥𝑖
0 > 𝑦𝑗0 + 𝜏 − ℙ 𝑦𝑗
0 > 𝑥𝑖0 + 𝜏
The net chance of a better outcome is then:
∆ = 1
𝑚 ∙ 𝑛 𝑠𝑖𝑗
𝑚
𝑗=1
𝑛
𝑖=1
The pairwise weight the analysis of an outcome of lower priority is: 𝜔𝑖𝑗= ℙ 𝑥𝑖
0 − 𝑦𝑗0 < 𝜏
16
The extended procedure taking into account ‘non-informative’ pairs
![Page 17: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/17.jpg)
Simulation study - Design
• Objective: To compare the standard and the extended procedures of generalized pairwise comparison
• Simulation of M = 1000 datasets of with N = 200 patients – One time-to-event outcome
– Threshold 𝜏 = 0 months
![Page 18: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/18.jpg)
HR HR HR
18
• Survival time: exponential distributions
Scenario 1 : Proportional hazards
Scenario 2 : Late treatment effect
Scenario 3 : early treatment effect
Simulation study - Design
![Page 19: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/19.jpg)
19
• Several treatment effect sizes – Hazard ratio 0,5; 0,7; 1
• Administrative censoring proportion – Uniform distribution
– Between 0% and 70%
Simulation study - Design
![Page 20: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/20.jpg)
20
• For each simulated dataset – Estimation of the net chance of a better outcome (standard and extended
procedure)
– Test of the null hypothesis (Permutation test, Log-Rank test)
• Endpoints – Bias
– Power
– Type 1 error
Simulation study - Design
![Page 21: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/21.jpg)
HR = 0,5
HR = 0,7
21
Scenario 1 – Proportionnal hazards
Péron, et al. SMMR 2016
![Page 22: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/22.jpg)
HR = 0,5
HR = 0,7
22
Scenario 1 – Proportionnal hazards
Péron, et al. SMMR 2016
![Page 23: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/23.jpg)
23
An explanation for this bias?
1,0
Surv
ival
Pro
bab
ility
0,0
Time
C𝐞𝐧𝐬𝐨𝐫𝐢𝐧𝐠 𝒚𝒋 E𝐯𝐞𝐧𝐭 𝒙𝒊
Standard procedure: U𝐧𝐢𝐧𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐯𝐞 → 𝑝𝑖𝑗= 0
Extended procedure: ℙ 𝑥𝑖
0>𝑦𝑗0+𝜏 = 1−
𝑆 𝐶 𝑥𝑖−𝜏
𝑆 𝐶(𝑦𝑗)
ℙ 𝑦𝑗0>𝑥𝑖
0+𝜏 = 𝑆 𝐶 𝑥𝑖+𝜏
𝑆 𝐶(𝑦𝑗)
U𝐧𝐢𝐧𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐯𝐞 𝐚𝐥𝐬𝐨 → 𝑝𝑖𝑗= 0
Treatment group
Control group
![Page 24: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/24.jpg)
24
A correction for this bias
𝑓𝑚 : Proportion of informative pairs for the trial 𝑚(𝑚 = 1,… ,𝑀)
∆ 𝑐𝑜𝑟𝑟𝑚 =∆ 𝑚
𝑓𝑚 : corrected net chance of a better outcome
HR=0,5
𝐵𝑖𝑎𝑠 = 𝔼 ∆ − ∆
Mean bias
Censoring rate
Péron, et al. SMMR 2016
![Page 25: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/25.jpg)
HR = 0,5
HR = 0,7
25
Scenario 1 – Proportional hazards
Censoring rate
Péron, et al. SMMR 2016
![Page 26: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/26.jpg)
26
Scenario 2 et 3 – Non Proportional hazards
Censoring rate Censoring rate
Po
we
r
Early treatment effect Late treatment effect
Type 1 error rate ≈ 5%
Péron, et al. SMMR 2016
![Page 27: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/27.jpg)
• Bias in the estimation of Δ – Less important
– Correction available when only one endpoint
• Power of the permutation test extended procedure > standard procedure – Exponential distribution
– Proportional hazards and administrative censoring < 67% (B Efron, Stanford
Univ, 1967)
– Late treatment effect
27
Conclusions of the simulation study
![Page 28: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/28.jpg)
Outline
• A patient-oriented measure of treatment benefit
• NCIC PA.3 trial => erlotinib in pancreatic cancer
28
![Page 29: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/29.jpg)
29
Probability for a random patient in the Treatment group to have a ‘better outcome’ than a random patient in the Control group …
Δ = ℙ 𝑿 > 𝒀 )
The net chance of a better outcome
Buyse M. Stat in med, 2010
Treatement group Control group
![Page 30: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/30.jpg)
30
Δ = ℙ 𝑋 > 𝑌 − ℙ(𝒀 > 𝑿)
Buyse M. Stat in med, 2010
Treatement group Control group
…minus the opposite probability.
The net chance of a better outcome
![Page 31: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/31.jpg)
31
ℙ(𝒀 = 𝑿)
Buyse M. Stat in med, 2010
Treatement group Control group
The net chance of a better outcome
![Page 32: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/32.jpg)
32
The net chance of a longer survival
Proportional hazards
Treatment group Control group
Time (months)
Net
ch
ance
of
a lo
nge
r su
rviv
al
Surv
ival
pro
bab
ility
Péron et al, JAMA oncology, 2016
![Page 33: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/33.jpg)
33
Proportional Hazards
Delayed treatment effect
Treatment group Control group
Treatment group Control group
Time (months)
Net
ch
ance
of
a lo
nge
r su
rviv
al
Surv
ival
pro
bab
ility
Time (months)
Net
ch
ance
of
a lo
nge
r su
rviv
al
Surv
ival
pro
bab
ility
Péron et al, JAMA oncology, 2016
The net chance of a longer survival
![Page 34: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/34.jpg)
34
Opposite hazards
Péron et al, JAMA oncology, 2016
Proportional Hazards
Treatment group Control group
Time (months)
Net
ch
ance
of
a lo
nge
r su
rviv
al
Surv
ival
pro
bab
ility
Treatment group Control group
Time (months)
Net
ch
ance
of
a lo
nge
r su
rviv
al
Surv
ival
pro
bab
ility
The net chance of a longer survival
![Page 35: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/35.jpg)
Simulation study - Design
• Objective: To assess the power of tests based on generalized pairwise comparisons for delayed treatment effect
• Simulation of M = 1000 datasets of with N = 200 patients – One time-to-event outcome
![Page 36: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/36.jpg)
36
Scenario 1 : Proportional hazards
Scenario 2 : Late treatment effect
Simulation study - Design
Survival
Time (months)
Survival
Time (months)
![Page 37: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/37.jpg)
37
• Administrative censoring proportion – Uniform distribution
– Between 0% and 40%
• For each simulated dataset – Estimation of the net chance of a better outcome (extended procedure) with
threshold 𝜏 0 to 20 months
– Test of the null hypothesis (Permutation test, Log-Rank test)
Simulation study - Design
![Page 38: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/38.jpg)
38
Delayed treatment effect - POWER
![Page 39: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/39.jpg)
39
Proportional Hazards - POWER
![Page 40: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/40.jpg)
When a long-term survival benefit is expected (anticancer immune therapy)
The net chance of a longer survival is: – Arguably more relevant than traditional methods
– More powerful than traditional method
40
Conclusions
![Page 41: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/41.jpg)
Outline
• Benefit-risk balance • NCIC PA.3 trial => erlotinib in pancreatic cancer
41
![Page 42: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/42.jpg)
42
The benefit-risk balance in the PA.3 trial
R
569 advanced pancreatic cancer
Gemcitabine + placebo Gemcitabine + erlotinib
284 285
Moore et al. JCO 2007
![Page 43: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/43.jpg)
43
The benefit-risk balance in the PA.3 trial
![Page 44: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/44.jpg)
Worst grade
related AE
Erlotinib group (n=282) Placebo group (n=280)
Grade 1 48 (17%) 69 (24.6%)
Grade 2 118 (41.8%) 89 (31.8%)
Grade 3 72 (25.5%) 47 (16.8%)
Grade 4 11 (3.9%) 6 (2.1%)
Grade 5 4 (1.4%) 3 (1.1%)
44
The benefit-risk balance in the PA.3 trial
![Page 45: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/45.jpg)
Worst grade
related AE
Erlotinib group (n=282) Placebo group (n=280)
Grade 1 48 (17%) 69 (24.6%)
Grade 2 118 (41.8%) 89 (31.8%)
Grade 3 72 (25.5%) 47 (16.8%)
Grade 4 11 (3.9%) 6 (2.1%)
Grade 5 4 (1.4%) 3 (1.1%)
45
The benefit-risk balance in the PA.3 trial
29,4% 18,9%
![Page 46: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/46.jpg)
46
Benefit-risk balance
Priority Erlotinib > Placebo Placebo > Erlotinib ∆∆[erlotinib]
1 : OS (Threshold = 2 months) 40.3 % 34.5 % 5.8 %
2 : Worst related AE grade 6.8 % 12.4 % -5.6 %
Global 47.1 % 46.9 % 0.2 % (P=.96)
Extended procedure
Péron J. Roy P. Ding K. Parulekar WR. Roche L and Buyse M. BJC 2015
![Page 47: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/47.jpg)
47
Benefit-risk balance
Péron J. Roy P. Ding K. Parulekar WR. Roche L and Buyse M. BJC 2015
![Page 48: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/48.jpg)
A package R corresponding to the standard procedure and to the extended procedure
• Available on CRAN (“BuyseTest”)
• Available on github (“https://github.com/bozenne/BuyseTest”)
48
Software implementation
![Page 49: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/49.jpg)
The net chance of a better outcome
– Is flexible
– Is meaningful
– Allows multicriteria analysis
– Can focus on long-term survival differences
– Is OK when hazards are not proportionals
– Is available
49
Conclusions
![Page 50: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/50.jpg)
Thank you
50
![Page 51: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/51.jpg)
𝑀𝑒𝑎𝑛 𝑠𝑞𝑢𝑎𝑟𝑒𝑑 𝑒𝑟𝑟𝑜𝑟 = 𝔼 ∆ − ∆2
51
Scénario 1 – Mean squared error
HR=0,5
Mean squared
error
![Page 52: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/52.jpg)
Methods – Pairwise comparisons
Let xi be the outcome of
i th subject in T (i = 1. … . n )
R
Control (C ) Treatment (T )
Let yj be the outcome of
j th subject in C (j = 1. … . m )
Yj Xi
favors T (favorable)
favors C (unfavorable)
pairwise comparison
Neutral or Uninformative 52
Buyse M. stat in med 2010
![Page 53: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/53.jpg)
Methods – Definition of thresholds
Coutinuous outcome
53 Buyse. stat in med 2010
Pair Rating
𝑥𝑖 𝑦𝑗 > Favorable
𝑥𝑖 𝑦𝑗 < ( ) Unfavorable
𝑥𝑖 𝑦𝑗 ≤ Neutral
𝑥𝑖 or 𝑦𝑗 missing Uninformative
![Page 54: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/54.jpg)
Methods – Definition of thresholds
Coutinuous outcome
54 Buyse. stat in med 2010
Pair Rating
𝑥𝑖 𝑦𝑗 > Favorable
𝑥𝑖 𝑦𝑗 < ( ) Unfavorable
𝑥𝑖 𝑦𝑗 ≤ Neutral
𝑥𝑖 or 𝑦𝑗 missing Uninformative
![Page 55: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/55.jpg)
Methods – Definition of priority
First priority outcome
Second priority outcome
Pair rating
Favorable NA Favorable
Unfavorable NA Unfavorable
Uninformative Favorable Favorable
Uninformative Unfavorable Unfavorable
Uninformative Uninformative Uninformative
55 Buyse. stat in med 2010
![Page 56: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/56.jpg)
Methods – Definition of priority
First priority outcome
Second priority outcome
Pair rating
Favorable NA Favorable
Unfavorable NA Unfavorable
Uninformative Favorable Favorable
Uninformative Unfavorable Unfavorable
Uninformative Uninformative Uninformative
56 Buyse. stat in med 2010
![Page 57: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/57.jpg)
Methods – Definition of priority
First priority outcome
Second priority outcome
Pair rating
Favorable NA Favorable
Unfavorable NA Unfavorable
Uninformative Favorable Favorable
Uninformative Unfavorable Unfavorable
Uninformative Uninformative Uninformative
57 Buyse. stat in med 2010
![Page 58: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/58.jpg)
Méthodes – Standard procedure for pairwise
scoring
Δ in named «chance of a better outcome»
58 Buyse. stat in med 2010
n
i
m
j
Uijnm
U1 1
1
otherwise 0
eunfavorabl is pair when the1
favorable is pair when the1
Y j,X i
Y j,X i
U ij
![Page 59: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/59.jpg)
Based on the Kaplan-Meier estimate of the survival function
𝕡 ( 𝑥𝑖0> 𝑦𝑗
0) 𝑥𝑖0> 𝑥𝑖 = −
𝑆 𝑇 𝑡
𝑆 𝑇 𝑥𝑖 𝑆 𝐶 𝑦𝑗𝑑𝑆 𝐶 𝑡
∞
𝑡>𝑦𝑗
𝑡∈ 𝑦𝑗𝜀𝑗=1
The extended procedure taking into account ‘non-informative’ pairs
0,5
1,0
Surv
ival
Pro
bab
ility
0,0
Time
0,8
Péron J et al, SMMR 2016
Patient i : censoring
Patient j : censoring
Treatment group
Control group
![Page 60: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/60.jpg)
60
𝕡 ( 𝑥𝑖0> 𝑦𝑗
0) 𝑥𝑖0> 𝑥𝑖 , 𝑦𝑗
0> 𝑦𝑗 = − 𝑆 𝑇𝑡𝑡 𝑡
𝑆 𝑇𝑡𝑡 𝑥𝑖 𝑆 𝐶𝑡𝑟𝑙 𝑦𝑗∙ 𝑆 𝐶𝑡𝑟𝑙 𝑡
+ − 𝑆 𝐶𝑡𝑟𝑙 𝑡−
∞
𝑡>𝑦𝑗
Efron, Berkeley Symp, 1967
0,5
1,0
0,0
When the estimation of the survival function is discontinue :
Surv
ival
Pro
bab
ility
Time
Patient i : censoring
Treatment group
Control group
Patient j : censoring
The extended procedure taking into account ‘non-informative’ pairs
![Page 61: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/61.jpg)
61
(𝜹𝒊, 𝜺𝒋) 𝒙𝒊 − 𝒚𝒋 > 𝝉 𝒙𝒊 − 𝒚𝒋 < −𝝉 𝒙𝒊 − 𝒚𝒋 < 𝝉
(𝟏, 𝟏) 1 -1 0
(𝟎, 𝟏) 1 𝑆 𝑇 𝑦𝑗 + 𝜏 + 𝑆 𝑇 𝑦𝑗 − 𝜏
𝑆 𝑇(𝑥𝑖)− 1
𝑆 𝑇 𝑦𝑗 + 𝜏
𝑆 𝑇 𝑥𝑖
(𝟏, 𝟎) 1 −𝑆 𝐶 𝑥𝑖 + 𝜏 + 𝑆 𝐶 𝑥𝑖 − 𝜏
𝑆 𝐶(𝑦𝑗) -1 −
𝑆 𝐶 𝑥𝑖 + 𝜏
𝑆 𝐶 𝑦𝑗
(𝟎, 𝟎)
1 −𝑆 𝐶 𝑥𝑖 − 𝜏 + 𝑆 𝐶 𝑥𝑖 + 𝜏
𝑆 𝐶 𝑦𝑗
− 𝑆 𝑇 𝑡 + 𝜏
𝑆 𝑇 𝑥𝑖 𝑆 𝐶 𝑦𝑗
∞
𝑡>𝑥𝑖−𝜏
𝑡∈ 𝑦𝑗𝜀𝑗=1
𝑑𝑆 𝐶 𝑡
− 𝑆 𝑇 𝑡 − 𝜏
𝑆 𝑇 𝑥𝑖 𝑆 𝐶 𝑦𝑗
∞
𝑡>𝑥𝑖+𝜏
𝑡∈ 𝑦𝑗𝜀𝑗=1
𝑑𝑆 𝐶 𝑡
− 𝑆 𝑇 𝑡 + 𝜏 + 𝑆 𝑇 𝑡 − 𝜏
𝑆 𝑇 𝑥𝑖 𝑆 𝐶 𝑦𝑗𝑑𝑆 𝐶 𝑡
∞
𝑡>𝑦𝑗
𝑡∈ 𝑦𝑗𝜀𝑗=1
− 1
− 𝑆 𝑇 𝑡 + 𝜏
𝑆 𝑇 𝑥𝑖 𝑆 𝐶 𝑦𝑗𝑑𝑆 𝐶 𝑡
∞
𝑡>𝑦𝑗
𝑡∈ 𝑦𝑗𝜀𝑗=1
− 𝑆 𝑇 𝑡 − 𝜏
𝑆 𝑇 𝑥𝑖 𝑆 𝐶 𝑦𝑗
∞
𝑡>𝑥𝑖+𝜏
𝑡∈ 𝑦𝑗𝜀𝑗=1
𝑑𝑆 𝐶 𝑡
−𝑆 𝐶 𝑥𝑖 + 𝜏
𝑆 𝐶 𝑦𝑗
Extension prenant en compte les temps jusqu’à
censure
Valeurs du score attribué à chaque paire en fonction d’un seuil τ représentant la différence minimale cliniquement significative
![Page 62: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/62.jpg)
62
proportion of pairs (%)
Priority Threshold Erlotinib >
Placebo
Placebo >
Erlotinib
∆erlotinib
1 : OS 6 months 16.8 13.1 3.6
2 : PFS 6 months 3.0 1.8 1.2
3 : Worst related AE grade 3 grades 2.3 5.8 -3.5
4 : OS 3 months 11.2 10.8 0.4
5 : PFS 3 months 3.4 2.7 0.7
6 : Worst related AE grade 2 grades 2.3 6.0 -3.7
7 : OS 0 months 9.5 9.1 0.4
8 : PFS 0 months 0.5 0.6 -0.1
9 : Worst related AE grade 1 grade 0.2 0.5 -0.3
Overall 49.2 50.4 -1.2 (P=.82)
Sensitivity analysis
![Page 63: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/63.jpg)
63
Hypothesis of the correction
1,0
0,0
Temps
Surv
ival
Pro
bab
ility
C𝐞𝐧𝐬𝐨𝐫𝐢𝐧𝐠 𝒚𝒋 E𝐯𝐞𝐧𝐭 𝒙𝒊
Treatment group
Control group
![Page 64: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/64.jpg)
proportion of pairs (%)
Priority Erlotinib > Placebo Placebo > Erlotinib ∆∆erlotinib
1 : OS (Threshold = 2 months) 37.0 32.3 4.7
2 : Worst related AE grade 7.5 15.7 -8.3
Overall 44.5 48.1 -3.6 (P=.51)
64
Main analysis of the benefit-risk balance of the erlotinib and gemcitabine
combination
Péron J. Roy P. Ding K. Parulekar WR. Roche L and Buyse M. BJC 2015
With the standard procedure
![Page 65: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/65.jpg)
65
Sensitivity analysis
Péron J. Roy P. Ding K. Parulekar WR. Roche L and Buyse M. BJC 2015
![Page 66: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/66.jpg)
66
Priority Erlotinib > Placebo Placebo > Erlotinib ∆∆[erlotinib]
1 : OS (Threshold = 2 months) 37.0 % 32.3 % 4.7 %
2 : Worst related AE grade 7.5 % 15.7 % -8.3 %
Global 44.5 % 48.1 % -3.6% (P=.51)
Standard procedure
Benefit-risk balance
Priority Erlotinib > Placebo Placebo > Erlotinib ∆∆[erlotinib]
1 : OS (Threshold = 2 months) 40.3 % 34.5 % 5.8 %
2 : Worst related AE grade 6.8 % 12.4 % -5.6 %
Global 47.1 % 46.9 % 0.2 % (P=.96)
Extended procedure
Péron J. Roy P. Ding K. Parulekar WR. Roche L and Buyse M. BJC 2015
![Page 67: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/67.jpg)
67
Extended procedure
Benefit-risk balance
Standard procedure
Péron J. Roy P. Ding K. Parulekar WR. Roche L and Buyse M. BJC 2015
![Page 68: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/68.jpg)
Priority FOLFIRINOX >
Gemcitabine
Gemcitabine >
FOLFIRINOX
∆∆[FOLFIRINOX]
1 : OS (Threshold = 2 months) 43.0% 21.7% 21.3%
2 : Worst related AE grade 8.2% 15.4% -7.2%
Global 51.2% 37.1% 14.0% (P=0.029)
68
PRODIGE 4 trial
Benefit-risk balance
![Page 69: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/69.jpg)
69
Priority Erlotinib > Placebo Placebo > Erlotinib ∆∆[erlotinib]
1 : OS (Threshold = 2 months) 37.0 % 32.3 % 4.7 %
2 : Worst related AE grade 7.5 % 15.7 % -8.3 %
Global 44.5 % 48.1 % -3.6% (P=.51)
PRODIGE 4 trial
NCIC PA-3 trial
Benefit-risk balance
Priority FOLFIRINOX >
Gemcitabine
Gemcitabine >
FOLFIRINOX
∆∆[FOLFIRINOX]
1 : OS (Threshold = 2 months) 43.0% 21.7% 21.3%
2 : Worst related AE grade 8.2% 15.4% -7.2%
Global 51.2% 37.1% 14.0% (P=0.029)
![Page 70: Analysing Survival Endpoints in Randomized Clinical Trials ...€¦ · Analysing Survival Endpoints in Randomized Clinical Trials using Generalized Pairwise Comparisons Dr Julien](https://reader035.fdocuments.net/reader035/viewer/2022070808/5f068dd77e708231d418905c/html5/thumbnails/70.jpg)
70
NCIC PA-3 trial